Commit Graph

2298 Commits

Author SHA1 Message Date
Miccah dbcb888063 Update Source interface to use SourceID and JobID types (#1774)
The previous implementation used int64 for both, which can be mixed up
easily. Using distinct types adds a layer of type safety checked by the
compiler.
v3.56.0
2023-09-14 11:28:24 -07:00
Mike Vanbuskirk bbc3be3b6c migrate buildpulse to integration test suite (#1775) 2023-09-13 15:25:12 -05:00
Mike Vanbuskirk b4329e0825 add buildpulse config to sources (#1764)
* add buildpulse config to sources

* remove tab indentation

* add correct repo ID

* wrap test command in gotestsum

* exclusion should be detectors

* fix exclusions to match community-test

* update tag to reflect comm. tests
2023-09-13 11:34:53 -05:00
Cody Rose 1155ee2736 Implement Gitlab source validation (#1765)
This PR implements validation of Gitlab source configuration.

I was hoping to be able to unify more of the implementation of Validate and Chunks, but there was more divergence than I expected. Specifically, Chunks handles a fair number of Gitlab errors that aren't configuration errors (e.g. "Gitlab returned a repo with an unparseable URL"). Accommodating these in the Validate code path felt wrong, and I wasn't able to create a common code path that could accommodate both Validate and Chunks without looking awful.
2023-09-13 11:51:12 -04:00
martinohmann 31d17c4f93 fix: add missing error check in archive handler (#1770)
Fixes #1769

The existing error check `errors.Is(err, archiver.ErrNoMatch) && depth >
0` only conditionally handled a specific error.

Any other error case was not short circuited and ended up causing a
nil-pointer dereference further down the method when `format.Name()` was
invoked.
2023-09-13 07:07:40 -07:00
Miccah 72b6a9ec6b Add a SourceType constant to all source packages (#1768) 2023-09-12 17:23:25 -07:00
Miccah be4d0bcb41 Refactor SourceManager to remove Enrollment (#1740)
* Refactor SourceManager to remove Enrollment

Initializing the Source will be the responsibility of the caller. The
SourceManager exposes a GetIDs method for getting a source and job ID.

* Update tests

* Update engine usage

* Update apiClient interface to have one GetIDs method

* Update SourceManager usage in engine
2023-09-12 16:58:38 -07:00
âh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d 91340c1e00 updating sendbirdorganizationapi detector to use tri-state verification (#1763) 2023-09-12 10:48:11 -04:00
Mike Vanbuskirk de540652cb verbosity updates to s3 source (#1750) v3.55.1 2023-09-11 14:53:43 -05:00
ahrav 90a07f0352 [chore] - fix slackwebhook detector (#1761)
* fix slackwebhook detector.

* sort imports.

* add test for 400.
2023-09-11 12:48:40 -07:00
Dustin Decker 72b3fa31a3 Improve private key detector (#1760)
* Surface extra data and check private keys directly against gitlab and github

* fix encrpypted private key test

* implement feedback

* mod tidy

* fix change

* Set timeout for SSH connections
v3.55.0
2023-09-11 12:05:27 -07:00
Zubair Khan 3f84a6700e add tri state verification to slack (not slack webhook) (#1731)
* add client, add known false positive check

* fix idiosyncracies

* cleanup

* cleanup comment

* add unexpected error test case

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2023-09-11 08:38:43 -07:00
ahrav fdeccf06a0 cache dupes w/ different decoders (#1754)
* only cache dupes that have different decoders.

* add test.

* remove file.

* update comment.
2023-09-11 08:18:48 -07:00
ahrav 70cdff915b add test for custom providers. (#1759) 2023-09-11 08:18:34 -07:00
Cody Rose 2c32b14df1 always close aws response body (#1758) 2023-09-11 10:42:14 -04:00
ahrav e53a72abd2 [chore] - Sentry detector update (#1746)
* add test case for account deactivated.

* Handle empty case.
2023-09-11 07:26:09 -07:00
Cody Rose 62ce9bac8b Retry AWS verification 403s (#1757)
This PR introduces retries on 403s in the AWS detector in attempt to work around erroneous SignatureDoesNotMatch errors. As part of the work, the detector has been refactored somewhat, resulting in two minor semantic changes:

Errors crafting the verification HTTP request no longer result in the candidate secret being discarded.
The known-words-based false positive check now runs (and potentially discards candidate secrets) even if verification is disabled. This unifies its behavior with the hash-based false positive check.
v3.54.4
2023-09-08 15:32:53 -04:00
trufflesteeeve e68ace5338 Always attempt to return a git link (#1756) 2023-09-08 15:17:29 -04:00
Marwan Sulaiman 3aa5369608 Add Tailscale detector (#1719)
* Add tailscale detector

* PR feedback: match on first element
2023-09-07 19:11:17 -07:00
ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d 8d66fde6de updating sendgrid detector to use tri-state verification (#1735)
* updating sendgrid detector to use tri-state verification
2023-09-07 14:21:03 -04:00
ahrav 2a9f34962d Add optional param to Chunks (#1747)
* Add interface for targeted chunking.

* use optional args.

* update Chunks method signature.

* update tests.

* fix test.

* update QueryCriteria type.
2023-09-07 09:03:37 -07:00
ahrav f6512ac4ca Use common chunker for archive handler (#1717)
* optimize the ReadToMax.

* add comment.

* remove dumb comment.

* update comment.

* fix test.

* lint.

* Expired invite link fix (#1713)

* Use comon chunker for archive handler.

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-09-06 09:26:33 -07:00
Harmon Herring bf581ae9f9 Fix pagerdutyapikey Detector (#1749)
* Fix pagerdutyapikey detector by broadening regex

* Add 'pd' to list of pagerdutyapikey keywords
2023-09-06 09:15:39 -07:00
ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d d3e7c5acc2 updating jiratoken and jiratokenV2 to use tri-state verification + updating tests (#1744) 2023-09-05 16:32:05 -04:00
ahrav abb131e502 [chore] - update Docker source (#1708)
* Add concurrency and common chunker.

* lint.

* address comments.
2023-09-05 07:40:38 -07:00
ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d b28a2b9177 updating sendbird detector to use tri-state verification (#1737)
* updating sendbird detector to use tri-state verification
2023-09-05 10:39:05 -04:00
Cody Rose afe708519b Validate S3 source (#1715)
This PR adds S3 source validation. This is accomplished by factoring out common "bucket visiting" logic to be used by both scanning and validation.
2023-09-05 10:18:58 -04:00
ahrav c9e6086644 Correctly create azure git links. (#1743) v3.54.3 2023-09-01 10:38:51 -07:00
ahrav 000065b225 [chore] - fix sentry detector (#1738)
* fix sentry detector to check response.

* use err.

* address comments.
2023-09-01 10:33:21 -07:00
s.shivasurya 0a949d7131 iterating on suggestions (#1742) 2023-09-01 10:20:18 -07:00
Mike Vanbuskirk 2e4b17d3f4 update jira detector to match new variable tokens (#1720)
* update jira detector to match new variable tokens

* add versioned interface

* use _v2 format for naming packages w. versioner

- also added documentation for internal/external contrib.

* migrate jira and jira_v2 secrets tests to newer version

* add v2 specific domain and email

* add support for tri-state verification

---------

Co-authored-by: Zubair Khan <zkhan124@umd.edu>
v3.54.2
2023-09-01 12:14:36 -04:00
Zubair Khan 78bbb89a30 add tri-state verification for twilio detector (#1729)
* add tri state for twilio

* save progress

* fix twilio tristate test

* resolve lint issue
2023-08-31 12:06:18 -04:00
s.shivasurya 6695cf1dce added sourcegraph token verification detection (#1730) 2023-08-31 08:47:13 -07:00
Miccah 4aa91337e5 Update to Go 1.21 (#1733)
* Update to Go 1.21

* go mod tidy
2023-08-31 08:46:43 -07:00
Zubair Khan 07702ea06d update slack webhook with tri-state verification (#1724)
* add tristate basics

* update test

* cleanup

* fix leading space

* fix accidental comment

* save changes

* update secret in gsm, fix bug

* fix linter issue

* cleanup

* use defaultClient
2023-08-30 18:18:17 -04:00
Cody Rose a2c0abbfd6 Unify S3 client creation logic (#1657)
This PR unifies some code paths within the S3 source. This is being done to better support a future implementation of S3 source validation; less code that runs means less code to validate. The logical change is to move the handling of "role-less" operation down the call tree, which allows for a single code path for more of the S3 code.

This PR also fixes a bug that would occur in the (rare) case that the source couldn't create a regional S3 client. Before, an error would be logged, but it would be followed by a panic. Now the bucket in question is skipped.
2023-08-30 17:49:37 -04:00
Miccah 522b2fab29 Add a cancel cause to job cancellation (#1728) 2023-08-30 12:00:44 -07:00
Miccah c77c1172c8 Add the 'Cause' family of functions to the context wrapper library (#1725)
Go 1.20 introduced `WithCancelCause`, `WithTimeoutCause`, and
`WithDeadlineCause` to allow adding a reason to context cancellations.
Adding it to our wrapper will allow us to use these features.
2023-08-30 07:57:45 -07:00
ahrav d6afca682b remove fmt.Print (#1727) v3.54.1 2023-08-30 07:17:38 -07:00
ahrav 4dc5eb7912 Optimize read to max (#1714)
* optimize the ReadToMax.

* add comment.

* remove dumb comment.

* update comment.

* fix test.

* lint.

* address comments.

* use limit reader.

* update equality check.

* update test.'

* use custom limit reader.

* address comments.

* revert fun.
2023-08-29 17:31:40 -07:00
Miccah 7ba880f47a Add AvailableCapacity method to SourceManager (#1665) 2023-08-29 12:36:44 -07:00
ahrav 2b1b1b5ad0 Add jobID to chunk. (#1721) 2023-08-29 12:02:30 -07:00
ahrav c51e8f8af5 buffer channel. (#1718) 2023-08-28 18:08:31 -07:00
Zubair Khan 66271aed7b add detectors that were missed (#1716) 2023-08-28 18:02:11 -07:00
Zachary Rice d1b1ac408e Expired invite link fix (#1713) v3.54.0 2023-08-28 14:21:13 -05:00
ahrav 0932ea224b [chore] - Prevent nil deref panic (#1709) 2023-08-26 20:39:50 -07:00
Miccah 5eb776cd61 Support cancelling a run from a JobProgressRef (#1663) 2023-08-25 10:43:33 -07:00
Cody Rose 33eed42e17 Test S3 role assumption (#1655)
This PR adds a test of the S3 role assumption functionality. It currently only tests role assumption within a single account.
2023-08-25 11:30:08 -04:00
Miccah 61977412df Add SourceName to JobProgressRef (#1664) 2023-08-25 07:48:25 -07:00
ahrav 4f4a79f62b Support azure git links (#1662)
* Support azure git links.

* update comment.

* update test names.
2023-08-24 14:36:52 -07:00