Commit Graph

80 Commits

Author SHA1 Message Date
Miccah ef9488c77d [chore] Log git output on error (#1180) 2023-03-15 15:32:29 -05:00
Tim Walter a7abd6231d Fix git commit date string formatting (#1181) 2023-03-14 22:39:12 -05:00
ahrav aa47e5e248 Only scanned staged git changes. (#1143) 2023-03-01 08:58:36 -08:00
Miccah 6209a80ce1 [chore] Address more linter errors (#1134)
* Address lint errors in detectors

* Update deprecated ioutil call
2023-02-28 10:00:41 -06:00
Miccah 4efe5313f4 [chore] Address lint errors (#1133)
* Update strings.Title to cases.Title

* Migrate go-genproto to google-cloud-go

See: https://github.com/googleapis/google-cloud-go/blob/main/migration.md

* Check error in test

* Check error from sem.Acquire

* Remove unused code
2023-02-27 21:03:47 -06:00
Miccah 0ce72ccda3 [chore] Remove logrus from github source (#1086)
* [chore] Remove logrus from github source

* Fix handleRateLimit test

* Fix tests
2023-02-09 18:02:04 -06:00
Miccah 1f0fd91205 Skip repo and continue scanning when encountering an error (#1080) 2023-02-08 11:33:01 -06:00
Bill Rich af6e3f8fdf Pull gitparse config options out of pkg consts (#1072)
* Pull gitparse config options out of pkg consts.

* Adjust naming
2023-02-04 13:19:23 -08:00
Bill Rich 00ebb2ed64 Full git log when targeting base merge commit (#1044)
* Full git log when targeting merge commits

* Full log is needed whenever base is specified.
2023-01-26 09:17:54 -08:00
Yassine Ilmi d720c0c0f3 Switch to retryableHttpClient for GitHub AuthN API Client + More Logs (#995)
* Adding missing flags to Readme

* Use retryableHttpClient by default for GitHub

* Adding repoUrl for scanning time log

* Use WithField instead of WithFields

* Updating README with lasted --help output
2023-01-09 09:21:56 -08:00
ahrav 009756dce6 add proto that was missing. (#986) 2022-12-23 13:27:07 -08:00
Miccah 7ac7fdae44 Add more logging for git sources (#974) 2022-12-13 17:51:57 -06:00
Bill Rich 33d32d2de4 Don't scan the --since-commit target (#960) 2022-12-06 13:24:27 -08:00
Bill Rich f1ec9e74eb Close files to clean up tmp files (#940) 2022-11-22 13:13:34 -08:00
Bill Rich 79cae3b82b Add newlines when file is split (#937) 2022-11-22 09:01:39 -08:00
Miccah 86f9e1288f Initialize scan options if given a nil pointer (#924) 2022-11-15 17:01:59 -06:00
ahrav dd141fb55f [oc-147] - Add context to all git methods (#901)
* Add context to all git methods.

* remove logrus.

* Add ctx.

* Address comments.

* Add error to clone failing.

* Return error.
2022-11-03 16:36:52 -07:00
Bill Rich 965279421c Support common ssh repo format (#878)
* Try ssh repo format

* Add tests
2022-10-28 11:56:03 -07:00
Bill Rich ab71b93f7d Add context to handler (#877)
* Add context to handler

* Return rather than break out of select
2022-10-28 08:57:55 -07:00
Bill Rich d7d614cc5f Copy buffer bytes (#864) 2022-10-25 09:09:47 -07:00
Bill Rich 958266ea84 Run chunker in pipeline (#859)
* Run chunker in pipeline

* Move ChunkSize and PeekSize to source package.

* Use new Chunk and Peek size location
2022-10-24 13:57:27 -07:00
Bill Rich 3d5f697f9a Use line aware chunking for git. (#858) 2022-10-24 13:00:03 -07:00
Dustin Decker 64ace363af Change commit to trace level logging 2022-10-24 08:59:52 -07:00
Miccah 2bc4985061 Add SSH config option for the git source (#830)
* Add SSH config option for the git source

The auth message is empty since we use the git binary underneath to
handle the SSH authentication.

* Import digitaloceanv2
2022-09-28 20:40:01 +02:00
Dustin Decker 752c848640 Show clone path for git repos (#823) 2022-09-26 14:36:55 -07:00
Bill Rich e3107ad6bb Move head and base normalization to source (#818) 2022-09-23 08:58:45 -07:00
Bill Rich 509cf8b6fa Use headref and check empty commits for base (#815) 2022-09-21 16:04:01 -07:00
Bill Rich 912d8e461d Add context so to avoid splitting creds. (#791)
* Add context so to avoid splitting creds.

* Add context newlines to expected results
2022-09-09 15:00:33 -07:00
ahrav 7ba583ca40 [THOG-681] - Handle errors sources (#783)
* Handle errors w/ github source.

* Fix loop var captured by func literal.

* Fix loop var captured by func literal.

* Set completed progress if the scan completes with no errors.

* Set progress to 100% if the scope and iteration are both 0.

* Fix commentary.

* Fix test.

* Return after the defer to os.RemoveAll.

* Fix unauth scan.

* Inline range loop.

* update tests for partial scan completion with errors. Ensure correct progress is set.

* Update progress for all sources.

* Update github test.

* Address comments.
2022-09-07 19:40:37 -07:00
Bill Rich 41936169c7 Use gitparse for unstaged changes. (#775) 2022-09-03 18:01:36 -07:00
Dustin Decker fa9479100e Add common sentry recover library and add into goroutines (#738)
* Add common sentry recover library and add into goroutines

* fix nits
2022-08-29 11:45:37 -07:00
Haz 4cc3529bc5 Added support for SSH URIs (#725) 2022-08-23 16:34:34 -07:00
Bill Rich a0d44a39f1 Use trufflesec git parser (#729)
* Use trufflesec git parser.

* wip

* Fix line numbers and linter feedback
2022-08-23 13:29:20 -07:00
Bill Rich 4a93e49eea Support scanning binary files in git sources (#684)
* Scan binary files for git sources

* Create data chunks in for loop

* Linter feedback and newline commit result

* Use disk buffered reader and chunker function
2022-08-10 16:10:45 -07:00
ahrav 30ebe84e3e [THOG-608] - Fix linter errors. (#701)
* Fix linter errors.

* Fix gist adding test.

* Update test string for mock JSON reply.

* Remove if.
2022-08-09 19:20:02 -07:00
ahrav 21e1ff4a8a Fix the order to correctly match the params in NewGit. (#676) 2022-07-28 13:23:45 -07:00
trufflesteeeve 176552b07a Fix commit attribution, git tests, and run make protos (#667)
* Update dependency to fix commit attribution, fix git tests

* Run make protos to match code with current proto definitions
2022-07-25 11:44:15 -04:00
trufflesteeeve 96106563a9 Remove git fragment trace (#656)
The fragment trace was a bit too verbose even at the trace level. We may
want to trace the file being chunked or something like that, but not the
entire diff.
2022-07-14 13:13:23 -04:00
trufflesteeeve e123e9f177 Cleanup individual repositories after scanning (#614) 2022-06-10 14:00:50 -04:00
Miccah fc18a5ae0c Bug fix and add authentication in shallow clone (#595) 2022-05-31 20:45:28 -05:00
Miccah 67ad2f2247 Shallow clone if --since-commit is provided (#564)
* Shallow clone if --since-commit is provided

* Set the user before constructing args

* Fix vbout detector

* Address PR comments

* Use a better name for timestamp
* Use net.URL.String method for the remote path
2022-05-24 10:49:03 -05:00
ahrav d2605354fe [THOG-332 ]Remove TokenSource interface from the init method of Source. (#539)
* Remove TokenSource interface from the init method of Source.

* Remove proto message.

* Remove proto message.

* Fix tests.

* Fix filesystem test.
2022-05-13 14:35:06 -07:00
ahrav b0d79180f6 [THOG-314] Add new parameter to the Init method for the source interface. (#529)
* Add new parameter to the Init method for the source interface.

* Add Oauth Token service.

* remove .test file.

* remove .test file.

* Fix param spelling.

* fix tests with new param in init

* Add missing gock lib.
2022-05-10 11:11:43 -07:00
ahrav e12432cef8 [THOG-315] Replace bytes.buffer with strings.builder. (#533)
* Replace bytes.buffer with string.builder.

* Remove profiling.

* Remove detector changes.

* ignore .test files.

* fix detectors removed.
2022-05-09 17:02:46 -07:00
ahrav 1bd5e9d9d2 fix merge conflicts. (#497) 2022-05-02 15:04:05 -07:00
ahrav eb9d3e333c [THOG-269] Increase test coverage for git. (#496) 2022-05-02 12:47:38 -07:00
ahrav 865bc5baf6 [THOG-248] fix broken gitlab tests (#437)
* Fix broken gitlab test.

* Close chunks chan from blocking indefinitely.

* Range over chunksCh in the event chunksCh is nil we don't run into an invalid memory address error.
Update warnings and move clone output information back.

* Remove commented out code.

* Remove .Run() because .CombinedOutput() should call .Run()

* Update test to include count check.

* Address PR comments.

* Fix merge issue.
2022-04-21 18:18:07 -07:00
Bill Rich d4c33918f6 Ignore safedir when scanning git repo (#438) 2022-04-21 10:02:20 -07:00
Dustin Decker 879e65b5c5 Handle nil file channel (#417) 2022-04-16 15:41:51 -07:00
Bill Rich 09de492ec9 Check that required info is present (#415) 2022-04-15 16:32:16 -07:00