Commit Graph

121 Commits

Author SHA1 Message Date
ahrav 13da76d357 skip files we can't scan (#2170) 2023-12-04 13:37:11 -08:00
ahrav 279f915799 [chore] - fix error comparisons (#2142)
* fix error comparisons

* fix imports
2023-12-01 08:32:41 -08:00
ahrav d334b3075e move all Git setup into Init method (#2105)
* add proto fields for git

* add uri to proto

* move all git setup into Init method

* fix logic for when to use repoPath
2023-11-16 13:59:53 -08:00
Miccah 9d6bc8c504 Refactor git source to support scanning units (#2083) 2023-11-01 09:52:58 -07:00
Miccah 52600a897a [chore] Replace chunks channel with ChunkReporter in git based sources (#2082)
ChunkReporter is more flexible and will allow code reuse for unit
chunking. ChanReporter was added as a way to maintain the original
channel functionality, so this PR should not alter existing behavior.
2023-11-01 09:22:44 -07:00
Mike Vanbuskirk 4636dc08f6 Add temp directory management (#1878)
* adds func to get scannerPIDs

* add cleanup and call to get pids

* move pid handling to git module

* remove PID logic from main

* refactor testing code to handle different exec name

* cleanup linting errors

* add better logging, fix dir if clause

* some PR fixups

* mod fixup

* add interfaces for helper funcs

* refactor cleanup into main, getPID into git

* lint and test fixups, remove fail on n<2 pids

* simplify pid sorting

* use filepath.Join

* use Args[0] for exec name, fix logger

* formatting fixup

* move functionality into cleantemp pkg

* go mod fixup

* remove redundant testing comment

* fix go.sum issues

* add 15m ticker loop for cleanup

* enclose ticker in function for goroutine defer

fix cleantemp interface

* make time more readable

* add check for non-local Trufflehog PIDs

* allow deletion even if no non-local pids found

* bundle intial cleanup into runCleanup func

* add explicit regex check for tempdir format
2023-10-26 12:28:56 -04:00
Bill Rich c5efa870ff Use latest dbr (#1955) 2023-10-24 07:52:49 -07:00
ahrav c4bc8fc7fa [bug] - correctly check err (#1824)
* correctly check err.

* address comments.

* update.

* add comment.

* update comment.
2023-09-27 15:52:07 -07:00
Miccah dbcb888063 Update Source interface to use SourceID and JobID types (#1774)
The previous implementation used int64 for both, which can be mixed up
easily. Using distinct types adds a layer of type safety checked by the
compiler.
2023-09-14 11:28:24 -07:00
Cody Rose 1155ee2736 Implement Gitlab source validation (#1765)
This PR implements validation of Gitlab source configuration.

I was hoping to be able to unify more of the implementation of Validate and Chunks, but there was more divergence than I expected. Specifically, Chunks handles a fair number of Gitlab errors that aren't configuration errors (e.g. "Gitlab returned a repo with an unparseable URL"). Accommodating these in the Validate code path felt wrong, and I wasn't able to create a common code path that could accommodate both Validate and Chunks without looking awful.
2023-09-13 11:51:12 -04:00
Miccah 72b6a9ec6b Add a SourceType constant to all source packages (#1768) 2023-09-12 17:23:25 -07:00
ahrav 2a9f34962d Add optional param to Chunks (#1747)
* Add interface for targeted chunking.

* use optional args.

* update Chunks method signature.

* update tests.

* fix test.

* update QueryCriteria type.
2023-09-07 09:03:37 -07:00
ahrav 2b1b1b5ad0 Add jobID to chunk. (#1721) 2023-08-29 12:02:30 -07:00
ahrav 4f4a79f62b Support azure git links (#1662)
* Support azure git links.

* update comment.

* update test names.
2023-08-24 14:36:52 -07:00
Bill Rich 2d2595a2e3 Move commits_scanned to ScanRepo (#1610) 2023-08-07 14:28:57 -07:00
ahrav 13999227b9 Use common chunk reader (#1596)
* Add common chunker.

* add comment.

* use better config name.

* Add common chunk reader to s3.

* Add common chunk reader to git, gcs, circleci.

* revert gcs.

* revert gcs.

* fix chunker.

* revert gcs.

* update cancellablewrite.

* revert impl.

* update to remove totalsize.

* Fix my goof.

* Use unified struct in chunkreader.

* return err instead of logging and returning.

* rename error to err.

* only send single ChunkResult even if there is an error and chunkBytes.

* fix logic.
2023-08-07 12:55:28 -07:00
Miccah e322c4b29d Fix nil pointer dereference to git ScanOptions (#1603) 2023-08-03 12:07:24 -05:00
Savely Krasovsky d062834997 initial support for bare repositories (#1499)
* feat: initial support for bare repositories

* feat: use concatenation instead of formatting and os.Getenv instead of os.Environ

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: go-git update with pre-receive hooks fix

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: remove info about pre-receive hook from README.md for now

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: don't scan staged while using --bare option, fixes to make it work with the latest master

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: small refactor according to #1518

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

---------

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
2023-08-03 11:23:41 -05:00
Bill Rich c995e93dcc Add commits scanned to log (#1600)
* Add commits scanned to log

* Use atomic
2023-08-02 14:10:54 -07:00
Miccah 69021f59c5 Refactor git source to allow ScanOptions and use source in engine (#1518)
* Refactor git source to allow ScanOptions and use source in engine

Refactor the Chunks method of the git Source to call out to two helper
methods: scanRepos and scanDirs which scans s.conn.Repositories and
s.conn.Directories respectively. The only notable change in behavior is
that a credential is no longer necessary if there are no
s.conn.Repositories to scan.

* Preserve ScanGit functionality of not cleaning up temporary files
2023-08-01 09:52:02 -05:00
ahrav eb00d0d4e1 [bug] - fix data races (#1577)
* fix data race.

* Add test and fix additional data race.

* address comments.
2023-07-31 11:12:38 -07:00
Bill Rich f39303495a Add commitsScanned metrics (#1533)
* Add commitsScanned metrics

* Just keep commit count
2023-07-25 11:31:01 -07:00
ahrav 9e0a2e9ddd [chore] - Remove password info from log (#1528)
* Remove password info from log.

* update.

* one more.
2023-07-22 20:25:45 -07:00
Richard Gomez 1594fddf05 feat(git): include line in github & gitlab links (#1466) 2023-07-11 20:02:27 -07:00
Zubair Khan 4334af4d34 scan GitHub PR and issue comments (#1435)
* issue comment scanning

* save progress

* test

* test for pr comment and issue comment

* add pagination support

* linter stuff

* make linter happy

* remove debug log

* readd logging

* github issue resolved

* var const block and handle rate limit

* remove magic number

* make gitURLParse a public function to use more generally

* fix test bug

* make comment scanning OPT-IN
2023-07-11 15:13:33 -04:00
Zachary Rice 452734adc8 remove head from git diff command, rename unstaged to staged (#1439) 2023-06-29 15:33:30 -05:00
Zachary Rice 4a77688097 use stringer again for now (#1430) 2023-06-26 14:33:54 -05:00
trufflesteeeve 11bff81def Use url redaction in git (#1399)
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-06-26 13:56:08 -05:00
Miccah f3152b6885 Implement SourceUnitUnmarshaller for all sources (#1416)
* Implement CommonSourceUnitUnmarshaller

* Add SourceUnitUnmarshaller to all sources using

All sources, with the exception of git, will use the CommonSourceUnit as
they only contain a single type of unit to scan.

* Fix method comments to adhere to Go's style guide
2023-06-23 11:15:51 -05:00
dillonstreator 648ef3b52c fix spelling errors (#1413) 2023-06-21 07:15:28 -07:00
Dustin Decker 572cb0e5dc Loosen up version check for git 2023-06-01 12:17:48 -07:00
Dustin Decker 183037ab34 Check that git meets version requirements (#1373) 2023-06-01 09:41:06 -07:00
Dustin Decker c8944825de Surface missing git as an error during initialization (#1362) 2023-05-26 15:23:08 -07:00
ahrav 1da7720912 Replace context.TODO. (#1349) 2023-05-19 11:09:51 -07:00
Brendan Shaklovitz fad34d4dc6 git worktree scanning fix for #827 (#1315)
* Fix worktree scan by setting EnableDotGitCommonDir

* Change `PlainOpenOptions` to set `EnableDotGitCommonDir` to true.
  In every current usage of this function, it is on an already-cloned
  repository, so it should always be valid to have this set. By doing
  so, it should fix some issues with worktrees.

* Remove unused go.mod replace directives

* Remove replace directives for libraries that are not in use.
2023-05-09 08:00:47 -07:00
Bill Rich f2924f3061 Make sure context lines are properly handled (#1331)
* Make sure context lines are properly handled

* Fix git test to account for context change
2023-05-05 12:51:27 -07:00
Brendan Shaklovitz be4147a24e Output git timestamps as UTC times (#1323) 2023-05-03 11:47:00 -05:00
ahrav 67972683ea [chore] - format log msg (#1299)
* format log msg.

* snake.

* lowercase repo.
2023-04-27 17:14:00 -07:00
ahrav a2266b4e28 add additional logging (#1298)
* add additional logging.

* update test.

* remove continue.

* address comments.
2023-04-27 16:48:04 -07:00
Dustin Decker 97ce27153a []bytes were being logged as b64ed string (#1255) 2023-04-14 06:43:26 -07:00
Zachary Rice fb9ae75661 Support for exclude globs at the git log level (#1202)
* init

* seems to be working

* better comment

* rm conditional

* Add more context to exclude-globs description
2023-03-28 10:46:03 -05:00
Miccah ef9488c77d [chore] Log git output on error (#1180) 2023-03-15 15:32:29 -05:00
Tim Walter a7abd6231d Fix git commit date string formatting (#1181) 2023-03-14 22:39:12 -05:00
ahrav aa47e5e248 Only scanned staged git changes. (#1143) 2023-03-01 08:58:36 -08:00
Miccah 6209a80ce1 [chore] Address more linter errors (#1134)
* Address lint errors in detectors

* Update deprecated ioutil call
2023-02-28 10:00:41 -06:00
Miccah 4efe5313f4 [chore] Address lint errors (#1133)
* Update strings.Title to cases.Title

* Migrate go-genproto to google-cloud-go

See: https://github.com/googleapis/google-cloud-go/blob/main/migration.md

* Check error in test

* Check error from sem.Acquire

* Remove unused code
2023-02-27 21:03:47 -06:00
Miccah 0ce72ccda3 [chore] Remove logrus from github source (#1086)
* [chore] Remove logrus from github source

* Fix handleRateLimit test

* Fix tests
2023-02-09 18:02:04 -06:00
Miccah 1f0fd91205 Skip repo and continue scanning when encountering an error (#1080) 2023-02-08 11:33:01 -06:00
Bill Rich af6e3f8fdf Pull gitparse config options out of pkg consts (#1072)
* Pull gitparse config options out of pkg consts.

* Adjust naming
2023-02-04 13:19:23 -08:00
Bill Rich 00ebb2ed64 Full git log when targeting base merge commit (#1044)
* Full git log when targeting merge commits

* Full log is needed whenever base is specified.
2023-01-26 09:17:54 -08:00