An LRU cache has a more favorable failure mode than the channel. With
the channel, if the consumer stopped consuming metrics, scanning would
block. With the LRU cache, metrics will be dropped when space runs out
and a log message emitted.
The UnitHook tracks metrics per unit of a job, and emits them on a
channel once finished. It should work even if the Source does not
support source units.
* Add generic glob filter
* Make nil filters safe
* Include glob in error
* Use better example for exclude and include test
* Allow user to configure the ambiguous case
* Rename Pass to ShouldInclude and invert logic
* Test default *Filter and Filter have the same behavior of allow
* Add property based tests
* Remove configuration for the not found ambiguous case
* Detector-Competition-Feat: Add Klaviyo API Secret Detector
* fix(error): add s1.VerificationError and remove specific code check.
* fix(error): add s1.VerificationError and remove specific code check.
* Use placeholder as default if field left empty and is required
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Drop unused func
* uncomment azure source
* update wording
---------
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Fix bug in chunker that surfaces with a flaky passed in io.Reader
The chunker was previously expecting the passed in io.Reader to always
successfully read a full buffer of data, however it's valid for a Reader
to return less data than requested. When this happens, the chunker would
peek the same data that it then reads in the next iteration of the loop,
causing the same data to be scanned twice.
Co-authored-by: ahrav <ahravdutta02@gmail.com>
* Fix EOF error check
* Use io.ReadFull in Chunker
---------
Co-authored-by: ahrav <ahravdutta02@gmail.com>
* update gitlabv2 to tri-state
* updating secret to s1 to match convention
* consolidating both versions of the gitlab detector
* remove gitlabV2 references
* Delete temp.txt
delete test file (note: these are not real secrets)
* updating gitlabV1 detector to only work w/ v1 secrets, and v2 detector only w/ v2 secrets
* update package name and add to defaults
* cleanup nesting
* lowercase package names
* update v1 detector to explicitly ignore results with glpat
* nit
* update package name
This PR updates the S3 source to use explicitly configured credentials if they're available and follow the normal AWS credentials waterfall if they're not. This is irrespective of whether role assumption is configured. This changes the previous behavior, which was to use waterfall credentials only if role assumption was configured and explicitly configured credentials only when it was not.
* added PR and Issue body scanning; adjusted CLI args to fit
* removed print statement from debugging
* removed exclude-commits; adjusted CLI flags
* minor changes to match main branch
* fixing logic
* updating README for --issues and --prs
* updating secret regex to include underscores and updating tests to have secret and domain within 20char range of keyword
* updating myfreshworks detector to use tri-state verification
* Update protos image to use correct go version
* Update image tag to match version
* use bullseye
* update proto image
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
* Add ability to dynamically scale concurrently running sources
Refactor SourceManager to use a counting semaphore to allow for
dymanically changing limits. This complicated `Wait() error` which needs
to return the first error encountered. We previously got that for free
using `errgroup.Group`, however now we need to handle that ourselves.
`Wait()` needs to return an error for use in the engine to set the
correct exit code.
* Group third party imports together
* Add functionality to update a source's link in the metadata with the updated line number.
* update comment.
* add logic to engine.
* only update link for non empty links.
* add tests for bb.
The previous implementation used int64 for both, which can be mixed up
easily. Using distinct types adds a layer of type safety checked by the
compiler.
* add buildpulse config to sources
* remove tab indentation
* add correct repo ID
* wrap test command in gotestsum
* exclusion should be detectors
* fix exclusions to match community-test
* update tag to reflect comm. tests
This PR implements validation of Gitlab source configuration.
I was hoping to be able to unify more of the implementation of Validate and Chunks, but there was more divergence than I expected. Specifically, Chunks handles a fair number of Gitlab errors that aren't configuration errors (e.g. "Gitlab returned a repo with an unparseable URL"). Accommodating these in the Validate code path felt wrong, and I wasn't able to create a common code path that could accommodate both Validate and Chunks without looking awful.
Fixes#1769
The existing error check `errors.Is(err, archiver.ErrNoMatch) && depth >
0` only conditionally handled a specific error.
Any other error case was not short circuited and ended up causing a
nil-pointer dereference further down the method when `format.Name()` was
invoked.
* Refactor SourceManager to remove Enrollment
Initializing the Source will be the responsibility of the caller. The
SourceManager exposes a GetIDs method for getting a source and job ID.
* Update tests
* Update engine usage
* Update apiClient interface to have one GetIDs method
* Update SourceManager usage in engine
* Surface extra data and check private keys directly against gitlab and github
* fix encrpypted private key test
* implement feedback
* mod tidy
* fix change
* Set timeout for SSH connections
This PR introduces retries on 403s in the AWS detector in attempt to work around erroneous SignatureDoesNotMatch errors. As part of the work, the detector has been refactored somewhat, resulting in two minor semantic changes:
Errors crafting the verification HTTP request no longer result in the candidate secret being discarded.
The known-words-based false positive check now runs (and potentially discards candidate secrets) even if verification is disabled. This unifies its behavior with the hash-based false positive check.
* update jira detector to match new variable tokens
* add versioned interface
* use _v2 format for naming packages w. versioner
- also added documentation for internal/external contrib.
* migrate jira and jira_v2 secrets tests to newer version
* add v2 specific domain and email
* add support for tri-state verification
---------
Co-authored-by: Zubair Khan <zkhan124@umd.edu>
This PR unifies some code paths within the S3 source. This is being done to better support a future implementation of S3 source validation; less code that runs means less code to validate. The logical change is to move the handling of "role-less" operation down the call tree, which allows for a single code path for more of the S3 code.
This PR also fixes a bug that would occur in the (rare) case that the source couldn't create a regional S3 client. Before, an error would be logged, but it would be followed by a panic. Now the bucket in question is skipped.
Go 1.20 introduced `WithCancelCause`, `WithTimeoutCause`, and
`WithDeadlineCause` to allow adding a reason to context cancellations.
Adding it to our wrapper will allow us to use these features.
Detect Snowflake secrets (compound URI of account, username, password) and enrich Secret Result with account and databases that the secret has access to.
The source manager initialization function was defined as `sourceID`
followed by `jobID`, while the source initialization function is the
reverse. This is confusing and easy to mix up since the parameters are
the same type.
This commit adds a test to make sure the source manager initializes in
the correct order, but it doesn't prevent the library user to make the
same mistake. We may want to consider using different types.
* add exportable validate function for github
* update validator
* use the context
* gate to prevent panic
* wrap error with context
* wrap error with context for basic auth and unauth
* add role assumption for s3 source
* refactor role assumption to repeatable string
user can pass array of roles to assume
* refactor s3 chunks to handle passed roleARNs
* add role-session name
use timestamp to make dynamic
* add docstring for rolearn strings()
* make sure role ars are passed into source
* refactor role assumption functionality
break s3 bucket scanning into sep. function
* add log check on assume role
* fix role iteration
- Make sure s3 struct is populated with roles
- add separate new client instantiation for role-based access
- iterates through each role
* add comment
* protobuf revert for merge
* re-run make proto
* lint cleanup
* cleanup TODOs
* drop redundant switch case in assumerole client
* use less verbose 'ctx' designator
* breakout functionality from Chunks
- separate functions for:
- enumerating buckets to scan
- scanning objects within the buckets
* remake protobuf defs
* allow scan to continue on single bucket err
* add readme docs
* minor fixups
With the introduction of the SourceManager, the chunks channel became
private and read-only. This provides a method to write chunks into the
channel as we transition away from needing to do that.
* Use custom context for archive handler of specialized archives.
* fix arg.
* fix test.
* use re-reader.
* use re-reader.
* Update error and comments.
* Add better error handling.
* update.
* setup
* update time out case to return detector result
* fix
* remove unneeded comment
* remove debug print
* cleanup
* more robust error handling
* reflect new detector template changes
* fixes
* mark response body check err as indeterminate
* Add handler for .deb file formats.
* Add handler for .rpm file formats.
* update.
* move logic to general archive handler.
* update const.
* Add compile time guard.
* Remove redundant parens.
* Add checks to make sure we have the tools installed to extract arhives.
* Limit size of temp file for arhive reading.
* handle nested archives.
* add comment.
* use consistent name for tempEnv -> env
* fix handler fxn signature.
* added functionality to scan docker images with digests instead of tags
* cleaned import statement
* added unit test for baseAndTag parsing + remote digest scan
Verification of an Alchemy secret requires putting the candidate secret directly into a URL. This makes the URL potentially sensitive, and if the request fails, we don't want to save it anywhere that might inadvertently get logged elsewhere - like the resulting error message. (Despite verification failing, this error message is only saved if the failure is indeterminate, which means that the secret might actually be live.)
* Init attempt at tui with bubble tea.
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Add starting and source selection options
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Rewrite models into a state machine
* Update source descriptions
* Make subpages implement tea.Model
* Rename page0 and page1 to be more descriptive
* Adjust styling and adding color consts
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Add helper generic function to call Update and type cast
* Setup plumbing for source configuration page
* Use CLI introspection for source configuration (WIP)
* Experiment with table view
* Replace table with form fields
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Change 🔒 to 💸
* Copy components from soft-serve
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Copy styles from soft-serve
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Copy common from soft-serve
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Refactor into pages
This is still a WIP, but the main structure is there.
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Trying out selector for wizard intro
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Use selector with custom View
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Change Item to be an enum
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add link pages
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Update source select to use selector
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Delete source configure page and add blank tabs
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add tab placeholder pages for configurationi
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Added headers and style to each tab
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Update with new sources
* Remove kingpin attribute from SourceItem
* Add basic form field and source structuring
* Hookup git form fields with an underlying textinput component
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Update forms for git and github
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Add labels per text input
* Add sources and adjust styling
* add basic trufflehog configuration page
* Add skip button to textinputs component
* Emit and handle textinputs skip/submit button commands
* Don't quit when q is pressed on the sourceConfigurePage
* Build trufflehog command based on source config vals
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Build flags based on truffle config inputs
* Update summary section
* Add generated truffle fields
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* update summary to correctly print info
* Go back a page when escape key is pressed
* WIP run page list
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Allow running trufflehog from the run page
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add option to view help docs
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* comment out unused styles and remove unused types
* Capitalize H in TruffleHog
* remove unneeded fmt.Sprintf
---------
Co-authored-by: mcastorina <m.castorina93@gmail.com>
This PR implements tri-state verification in the FTP detector. The verification timeout was made injectable to support a new test case. Some test cases that had already been broken have been fixed as well.
* Add common chunker.
* add comment.
* use better config name.
* Add common chunk reader to s3.
* Add common chunk reader to git, gcs, circleci.
* revert gcs.
* revert gcs.
* fix chunker.
* revert gcs.
* update cancellablewrite.
* revert impl.
* update to remove totalsize.
* Fix my goof.
* Use unified struct in chunkreader.
* return err instead of logging and returning.
* rename error to err.
* only send single ChunkResult even if there is an error and chunkBytes.
* fix logic.
* Add SourceManager to Engine struct
* Update Engine methods to use the SourceManager
* Fix GCS test
The original was testing that `Init()` errors weren't surfaced in
`Finish()`, but the `SourceManager` changed that behavior.
* JobProgress race fixes
* Add contextual values
* Remove unused code
* Add debug logs
* Rename WithConcurrency to WithConcurrentSources
* Always forward chunks to the output chunks channel
This PR implements tri-state verification for the LDAP detector. This implementation looks for network errors to explicitly flag as indeterminate, rather than authentication errors to explicitly flag as determinate; this is because the error that occurs from authentication failures doesn't appear to have its own type and I didn't want to have to match on the error message text.
* feat: initial support for bare repositories
* feat: use concatenation instead of formatting and os.Getenv instead of os.Environ
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* fix: go-git update with pre-receive hooks fix
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* fix: remove info about pre-receive hook from README.md for now
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* fix: don't scan staged while using --bare option, fixes to make it work with the latest master
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* fix: small refactor according to #1518
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
---------
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* Github Oauth2 verification
* Use prefix and include RawV2
* Make gh_oauth2 a new detector
* Remove unused struct
* Remove versioner
* Remove unused code
* Refactor git source to allow ScanOptions and use source in engine
Refactor the Chunks method of the git Source to call out to two helper
methods: scanRepos and scanDirs which scans s.conn.Repositories and
s.conn.Directories respectively. The only notable change in behavior is
that a credential is no longer necessary if there are no
s.conn.Repositories to scan.
* Preserve ScanGit functionality of not cleaning up temporary files
* gitparse: Use an object for currentDiff instead of a pointer
* gitparse: Use an object for currentCommit instead of a pointer
* Revert "gitparse: Use an object for currentCommit instead of a pointer"
This reverts commit c5f0708b4a.
* Support fatal errors in job reports
* WIP: JobReporter and JobInspector
* WIP: JobReportHook and JobReportRef
* Add ChunkError type and asyncRun helper method
* Rename JobReport to JobProgress
* Return a closed channel from Done when the JobProgress is nil
* Comment catchFirstFatal function
It turns out that GetCallerIdentity returns a surprising quantity of transient, false-negative 403 responses that carry the SignatureDoesNotMatch error reason. I don't know why this is happening, but their transient nature makes them indeterminate verification failures and they should be flagged as such. The AWS detector has therefore been modified to specifically look for the InvalidClientTokenId error reason in 403 responses and mark all other responses as indeterminate.
In addition to the functional changes this PR contains some updates to the test code that allow us to test them.
#1454 modified one of the Github enumeration code paths in a way that broke an integration test by causing one client's transport to be used for the construction of a different client, causing authentication failures. This saves the original transport for use, fixing the test.
This PR makes the Alchemy detector run its known false positive check even if verification is disabled. This isn't the most important detector but it's the template for new ones so getting a good pattern nailed down is important.
Moving the check allowed me to rewrite the determinacy logic to hopefully be more clear.
* Miscellaneous SourceManager updates
* Own the chunks channel instead of accepting it as an input
* Add Chunks and Wait methods
* Fix bug in Enroll so it actually returns the handle
* Add context.Context parameter to the SourceInitFunc type
* Add SourceManager tests for Run and Wait methods
* Rename man variables to mgr
* Implement SourceManager basics
* Rename identifiers and add a default headlessAPI implementation
* Rewrite to use SourceInitFunc
* Update variable name to accurately reflect its value
Previously, the various JDBC detectors would independently try to verify credentials by a process of trying various permutations of candidates one-by-one. The upcoming tri-state verification work will need to add sophistication to this process in the same way for each one, so this PR first combines all of the logic so it can be upgraded in a single spot.
* saving progress
* proto changes
* run make protos
* verify response, add test case
* resolve linter warning about unescaped . in regex pattern
* resolve overlapping proto number
JDBC redaction could fail in some irritating edge cases involving passwords that contain the @ character. The logic has been tweaked to eliminate these cases and some tests have been added.
* issue comment scanning
* save progress
* test
* test for pr comment and issue comment
* add pagination support
* linter stuff
* make linter happy
* remove debug log
* readd logging
* github issue resolved
* var const block and handle rate limit
* remove magic number
* make gitURLParse a public function to use more generally
* fix test bug
* make comment scanning OPT-IN
* Add CancellableWrite helper function
* Create SourceUnitEnumerator interface and EnumerationResult struct
* Implement SourceUnitEnumerator for the filesystem Source
* Omit explicit zero values
* verify response body with expected keywords
* remove debug log
* add extra test case
* migrate from ioutil to io
* close body and only check for one keyword
* cleanup
* init
* add detector type
* rotate leaked credentials
* tighten up username pattern
* isolated prefixregex as overrriding new line stuff
* passwordPat working now
* add username test
* fix edge case
* cleanup
* make linter happy
* make linter happy rd 2
* skip error logging
* fix test
* add password regex helper func
* make test more robust
* cleanup PR
* remove comments
* clarify prepend rationale
* Exit with non-zero exit code on chunk source error
* Exit with a non-zero exit code whenever we hit an error getting
chunks. Previously the error would be logged but trufflehog would exit
with a 0 (success) status code.
* fix gcs test
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
Co-authored-by: ahrav <ahravdutta02@gmail.com>
* Implement CommonSourceUnitUnmarshaller
* Add SourceUnitUnmarshaller to all sources using
All sources, with the exception of git, will use the CommonSourceUnit as
they only contain a single type of unit to scan.
* Fix method comments to adhere to Go's style guide
* Add Validator interface and example
* Close sockets and improve error messages
* Remove duplicate error
* Use var declaration so err slice can be nil
* Use heuristic to choose the most likely UTF-16 decoded string
* Assume ASCII and include valid BE and LE bytes
* Remove unused code
* Assume ASCII and return nil when not utf16
---------
Co-authored-by: bill-rich <bill.rich@gmail.com>
* init
* look for client id and client secret, encode them for basis auth
* add tests
* test without checking the contents of response
* confirm access_token exists
* cleanup test
* explain in code that an undocumented grant_type is used
* remove use of deprecated ioutil, remove dead code, return errors instead of just logging
* directly pull access token
* update error text, remove redundant body close()
* import new detector into defaults
When a Result is emitted, it should include
the `chunk.Data []byte` so that we can utilize
the blob of data which caused the result.
This makes it so something catching the results
does not have to maintain a collection of chunks
to correlate the two together.
* Add Base64URLSafe decoder
* Add decoder that can decode base64 strings with '_' and '-' instead of
of '+' and '/'.
* Combine url-safe b64 decoder into b64 decoder
Generic API keys like shown in the example below is getting excluded:
api_key=9e107d9d372bb6826bd81d3542a419d6 because of following regex patterns:
\b[A-Fa-f0-9]{32}\b
\b[A-Fa-f0-9x]{6,99}\b
The base64 decoding logic is getting hit and NOT returning an error, and thus it continues thinking it is base64 decoded.
* Fix worktree scan by setting EnableDotGitCommonDir
* Change `PlainOpenOptions` to set `EnableDotGitCommonDir` to true.
In every current usage of this function, it is on an already-cloned
repository, so it should always be valid to have this set. By doing
so, it should fix some issues with worktrees.
* Remove unused go.mod replace directives
* Remove replace directives for libraries that are not in use.
* Fix error where some files do not get properly scanned due to order of
extraction / decompression steps. Doing decompression first ensures
that a compressed archive (e.g., gzipped zip file), is handled
correctly.
* Implement EndpointCustomizer
Add the EndpointCustomizer interface and EndpointSetter convenience struct,
implement EndpointCustomizer for github and gitlab detectors, and add
parsing, verification, and applying user-supplied configuration.
* Check error from SetEndpoints
* Rename variable for clarity
* add sharepoint source proto
* create sharepoint oss protos
* add email field, remove oauth2 type, update token to refresh_token
* rename path to link
* restore clientcredentials
* restore s3 and confluence proto changes from make command
* Add in-memory caching lib, used by the GCS source.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* Fix static check.
* Add test for NewWithData.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* update comment.
* update comments.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* remove unused dep.
* address comments.
* Add exists method.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* fix test.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* rebase.
* split encode resume by comma.
* Use a persistable cache.
* fix merge.
* fix merge.
* Add progress as part of the cache given it will be the persistence layer.
* Add test for making sure the cache doesn't persist when the increment value is not met.
* fix tests.
* Resolve#1167 by adding support for the AWS_SESSION_TOKEN environment variable and adding a --session-token cli arg
* fix error message
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
* Adjust types to use DetectorID struct
* Parse versions with detector include and exclude input
* Update detectors filter to use version
Co-authored-by: steeeve <steve@trufflesec.com>
* Implement Versioner for github, gitlab, and npm detectors
Co-authored-by: steeeve <steve@trufflesec.com>
---------
Co-authored-by: steeeve <steve@trufflesec.com>
* Refactor generating CustomRegex results into a helper function
* Added errGroup for createResults, and ensure goroutines are non-blocking
* clean return
---------
Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>
This will concatenate all errors together into a single string. When
possible, it would be better to log the actual errors slice to take
advantage of structured logging.
* Add ability to include and exclude detectors
* Trim space before checking for empty items
* Explicitly check for integer overflow
* Use strconv.ParseInt instead of strconv.Atoi
* Address comments
* Rename directories to paths
* Generate protos
* Add file scanning support to filesystem source
* Add directories back to filesystem proto
* Generate protos
* Combine paths and directories from in source
* Add filesystem filter
* Address comments
In this case just Heroku and LinearAPI. But this includes the Moonclerck
detector, which appears to be a typo that got turned into a separate
detector type.
Co-authored-by: zubairk14 <zubair.khan@trufflesec.com>
* Add Type() to detector interface
The goal here is to allow the detector type information to be used
without the need for reflection. This could possibly allow us to more
easily inject information into detectors or filter them out if
necessary.
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
* remove test detector
---------
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
Checking for false positives can lead to results being removed before
ever getting the opportunity to verify them. Users are already
responsible for verification of custom detectors, so let's not interfere
with how they choose to use it.
* Adding missing flags to Readme
* Use retryableHttpClient by default for GitHub
* Adding repoUrl for scanning time log
* Use WithField instead of WithFields
* Updating README with lasted --help output
* Allow using a glob for include list.
* Update command flag.
* Make comment more clear.
* update comment.
* Allow scanning repo and org at the same time.
* Remove verifying successRanges because it is unused in webhook
* Move custom_detectors validation code into its own file
* Initial implementation of custom regex detector
Secret verification is done via webhook.
* Add CustomRegex detector type
* Add upper bound to permutation
* Return early if the context is canceled
* Add headers from configuration
* Add detector name as a key in the JSON body
* Implement faster algorithm for productIndices
* Add validation skeleton
* Add custom detector validation with tests
* Validate and test regex vars
* Implement RegexVarString
* Use RegexVarString for validating regex variables
* Add numerics to the regex variable matching
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add custom_detectors proto
* Generate proto code
* Create custom_detectors package
Also create protoyaml package to test YAML unmarshalling the
configuration.
* Simplify custom_detectors proto by removing connection
* Generate proto code
* Update custom_detectors parsing tests
Fixing help subcommand reference to be `--help`, not `-h`, which does not work:
```
$ trufflehog git -h
trufflehog: error: unknown short flag '-h', try --help
$ trufflehog --version
trufflehog 3.16.4
```
* Fixed the brew install tap command
It seems you need to run the full tap, not just trufflehog.
Signed-off-by: JJ Asghar <awesome@ibm.com>
* Update README.md
Signed-off-by: JJ Asghar <awesome@ibm.com>
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
* simplify monogo pattern
* do URI verification locally, while attempting to defuse SSRF
* test SSRF defuse
* simplify err check logic per linter recommendation
* split up detectors
* address comments
* remove unused var
* Remove the check to filter and return only a single unverified result.
* Revert "Remove the check to filter and return only a single unverified result."
This reverts commit 494e432803.
* Add new CLI flag to filter unverified results.
Also, don't fetch a github user or their token when both are known. This
currently only affects the Github Token auth type. Github App
installations will continually fetch tokens every time we clone a repo.
In the future we should check the `ExpiresAt` field of the Github App
token and determine if we need to fetch a new one at that point.
* Add SSH config option for the git source
The auth message is empty since we use the git binary underneath to
handle the SSH authentication.
* Import digitaloceanv2
I'm not sure I fully understand why this issue exists. But I think the
short version is this: When we attempted to paginate users, we would set
a variable's Page value. But that variable appears to not actually be a
pointer, despite being added as one. It probably has to do with how
struct embedding works. Either way, if we make the overall options
variable the whole thing, and update its embedded struct with our page
variable, everything works out.
* [THOG-162] Implement JDBC verification for select drivers
Also includes integration tests for postgres and mysql via docker. To
run, execute the following (untested what will happen if the docker
images aren't installed):
go test -tags=detectors,integration ./pkg/detectors/jdbc
* Make jdbc regex a bit more strict
* Surface the context to allow the caller to set a timeout
* Update readme.go
Readme has change the way they issue their keys
now its like rdme_{70} ascii chars
* Update readme.go
* Update readme.go
* Update readme.go
The tester seems to be working fine with the new defaultclient code
* Handle errors w/ github source.
* Fix loop var captured by func literal.
* Fix loop var captured by func literal.
* Set completed progress if the scan completes with no errors.
* Set progress to 100% if the scope and iteration are both 0.
* Fix commentary.
* Fix test.
* Return after the defer to os.RemoveAll.
* Fix unauth scan.
* Inline range loop.
* update tests for partial scan completion with errors. Ensure correct progress is set.
* Update progress for all sources.
* Update github test.
* Address comments.
* [THOG-643] Implement independent log level controls
There are two log level controls to mentally distinguish. Log levels
associated with a sink (e.g. stdout and streamed), and log levels
associated with a logger (e.g. a GitHub source).
The level is determined to be the minimum of the two. If a sink is at
level 0, then it will only output level 0 logs regardless of the
logger's level. This is best demonstrated by TestSinkWithName.
* Rename WithName to WithNamedLevel
* Check flush errors
* Replace IncreaseLevelCore with custom LevelCore
Adding a leveler that was less verbose would cause the initialization
fail, and therefore not be added to the core. This check is only at
the time of initialization.
An alternative approach to creating our own core is to set the child log
level equal to the parent, so initialization is guaranteed (with the
added benefit of intuitive behavior).
* Use controller if it exists, otherwise inherit parent's log level
* Cleanup some tests
* Fix the starting index value for plus line check.
* Set the correct source type for notifications.
* Reset old value.
* Fix the starting index value for plus line check.
* Fix len check.
* Reset old value.
* Add tests.
* Update tests.
* Update tests.
* Start updating detectors that have two part creds to record the raw result as ID + secret.
* Add more detectors.
* More detectors.
* More detectors.
* remove comment out imports.
* Scan binary files for git sources
* Create data chunks in for loop
* Linter feedback and newline commit result
* Use disk buffered reader and chunker function
* Use a config struct when scanning and engine source.
* fix tests.
* Move test_helpers to the sources pkg.
* Handle ScanGit error in tests.
* adderss comments.
* Use functional options.
* Remove temp var.
* Add better var names for the setup functions for each config.
* Remove unused var.
* fix error logs.
* fix error logs.
* single line.
* remove blank lines.
The fragment trace was a bit too verbose even at the trace level. We may
want to trace the file being chunked or something like that, but not the
entire diff.
* added common regex patterns for detectors
* For HexPattern
* enhancements
* used parseInt
* enhancement
* enhanced regex for email and subdomain
* enhancement for email pattern
* update pattern and detector
Co-authored-by: Roxanne Tampus <roxannetampus02@gmail.com>
* Remove the ending word boundary for the AWS dectector. This will prevent missing secrets that end with / due to it not being ASCII.
* Update regex to be more strict.
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
Please review the [Community Note](https://github.com/trufflesecurity/trufflehog/blob/main/.github/community_note.md) before submitting
### TruffleHog Version
<!--- Please run `trufflehog --version` to show the version. If you are not running the latest version, please upgrade because your issue may have already been fixed. --->
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
### Description
Please review the [Community Note](https://github.com/trufflesecurity/trufflehog/blob/main/.github/community_note.md) before submitting
## Description
<!--- Please leave a helpful description of the feature request here. --->
## Problem to be Addressed
<!--- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] --->
## Description of the Preferred Solution
<!--- A clear and concise description of what you want to happen. What
information may be required and what would be the preferred way to provide it?
### Preferred Solution
<!--- A clear and concise description of what you want to happen. What
information may be required and what would be the preferred way to provide it?
What should the output include? --->
## Additional Context
### Additional Context
<!--- Add any other context or screenshots about the feature request here. --->
### References
#### References
<!---
Information about referencing Github Issues: https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests
Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Vendor blog posts or documentation? For example:
Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
Most testing is handled automatically by our GitHub Actions workflows.
## Local GitHub Action Testing
In some cases you may wish to submit changes to the Trufflehog GitHub Action. Unfortunately GitHub does not provide a 1st-party testing environment for testing actions outside of GitHub Actions.
Fortunately [nektos/act](https://github.com/nektos/act) enables local testing of GitHub Actions.
### Instructions
1. Please follow [the installation instructions](http://https://github.com/nektos/act#installation) for your OS.
2. The first run of `act` will ask you to specify an image. `Medium` should suffice.
3. You'll need to configure a personal-access-token(PAT) with: `repo:status`, `repo_deployment`, and `public_repo` permissions.
4. Set an environment variable named `GITHUB_TOKEN` with the PAT from the previous step as the value: `$ export GITHUB_TOKEN=<your_PAT>`
5. Run the following command from the repository root: `act pull_request -j test -W .github/workflows/secrets.yml -s GITHUB_TOKEN --defaultbranch main`
6. If the job was successful, you should expect to see output from the scanner showing several detected secrets.
7. If you want to omit the context of a pull request event and just test that the action starts successfully, run: `act -j test -W .github/workflows/secrets.yml -s GITHUB_TOKEN --defaultbranch main`
[](/pkg/detectors) <!-- Badge must be run from staging, see badges/shields#5967 -->
+ All I see is `🐷🔑🐷 TruffleHog. Unearth your secrets. 🐷🔑🐷` and the program exits, what gives?
+ That means no secrets were detected
+ Why is the scan is taking a long time when I scan a GitHub org
+ Unauthenticated GitHub scans have rate limits. To improve your rate limits, include the `--token` flag with a personal access token
+ It says a private key was verified, what does that mean?
+ Check out our Driftwood blog post to learn how to do this, in short we've confirmed the key can be used live for SSH or SSL [Blog post](https://trufflesecurity.com/blog/driftwood-know-if-private-keys-are-sensitive/)
+ Is there an easy way to ignore specific secrets?
+ If the scanned source [supports line numbers](https://github.com/trufflesecurity/trufflehog/blob/d6375ba92172fd830abb4247cca15e3176448c5d/pkg/engine/engine.go#L358-L365), then you can add a `trufflehog:ignore` comment on the line containing the secret to ignore that secrets.
# :newspaper: What's new in v3?
TruffleHog v3 is a complete rewrite in Go with many new powerful features.
- We've **added over 700 credential detectors that support active verification against their respective APIs**.
- We've also added native **support for scanning GitHub, GitLab, filesystems, and S3**.
- We've also added native **support for scanning GitHub, GitLab, filesystems, S3, GCS and Circle CI**.
- **Instantly verify private keys** against millions of github users and **billions** of TLS certificates using our [Driftwood](https://trufflesecurity.com/blog/driftwood) technology.
- Scan binaries and other file formats
- Available as a GitHub Action and a pre-commit hook
## What is credential verification?
For every potential credential that is detected, we've painstakingly implemented programatic verification against the API that we think it belongs to. Verification eliminates false positives. For example, the [AWS credential detector](pkg/detectors/aws/aws.go) performs a `GetCallerIdentity` API call against the AWS API to verify if an AWS credential is active.
For every potential credential that is detected, we've painstakingly implemented programmatic verification against the API that we think it belongs to. Verification eliminates false positives. For example, the [AWS credential detector](pkg/detectors/aws/aws.go) performs a `GetCallerIdentity` API call against the AWS API to verify if an AWS credential is active.
The S3 source supports assuming IAM roles for scanning in addition to IAM users. This makes it easier for users to scan multiple AWS accounts without needing to rely on hardcoded credentials for each account.
The IAM identity that TruffleHog uses initially will need to have `AssumeRole` privileges as a principal in the [trust policy](https://aws.amazon.com/blogs/security/how-to-use-trust-policies-with-iam-roles/) of each IAM role to assume.
To scan a specific bucket using locally set credentials or instance metadata if on an EC2 instance:
Multiple roles can be passed as separate arguments. The following command will attempt to scan every bucket each role has permissions to list in the S3 API:
This project exists thanks to all the people who contribute. [[Contribute](CONTRIBUTING.md)].
@@ -204,16 +427,29 @@ This project exists thanks to all the people who contribute. [[Contribute](CONTR
</a>
## Contributing
# :computer: Contributing
Contributions are very welcome! Please see our [contribution guidelines first](CONTRIBUTING.md).
We no longer accept contributions to TruffleHog v2, but that code is available in the `v2` branch.
### Adding new secret detectors
## Adding new secret detectors
We have published some [documentation and tooling to get started on adding new secret detectors](hack/docs/Adding_Detectors_external.md). Let's improve detection together!
## License Change
# Use as a library
Currently, trufflehog is in heavy development and no guarantees can be made on
the stability of the public APIs at this time.
# License Change
Since v3.0, TruffleHog is released under a AGPL 3 license, included in [`LICENSE`](LICENSE). TruffleHog v3.0 uses none of the previous codebase, but care was taken to preserve backwards compatibility on the command line interface. The work previous to this release is still available licensed under GPL 2.0 in the history of this repository and the previous package releases and tags. A completed CLA is required for us to accept contributions going forward.
# :money_with_wings: Enterprise product
Are you interested in continuously monitoring your Git, Jira, Slack, Confluence, etc.. for credentials? We have an enterprise product that can help. Reach out here to learn more https://trufflesecurity.com/contact/
We take the revenue from the enterprise product to fund more awesome open source projects that the whole community can benefit from.
d="M 27.791,115.217 1.541,69.749 a 11.499,11.499 0 0 1 0,-11.499 l 26.25,-45.467 a 11.499,11.499 0 0 1 9.959,-5.75 h 52.5 a 11.5,11.5 0 0 1 9.96,5.75 l 26.25,45.467 a 11.499,11.499 0 0 1 0,11.5 l -26.25,45.466 a 11.499,11.499 0 0 1 -9.96,5.75 h -52.5 a 11.499,11.499 0 0 1 -9.959,-5.75 z"
id="b"/></defs><g
inkscape:label="Layer 1"
inkscape:groupmode="layer"
id="layer1"
transform="translate(-5.467857,-5.738096)"><g
id="g706"><path
fill="#f05133"
d="M 88.304634,18.167061 77.689818,7.5527709 c -0.611187,-0.611452 -1.602581,-0.611452 -2.214298,0 l -2.203979,2.204508 2.795852,2.7958521 c 0.649817,-0.219604 1.394883,-0.07223 1.912673,0.445823 0.520965,0.521229 0.667015,1.272646 0.441854,1.924844 l 2.695045,2.694781 c 0.652198,-0.224895 1.404144,-0.07938 1.924844,0.442119 0.727604,0.727604 0.727604,1.906588 0,2.634721 -0.728134,0.727869 -1.907117,0.727869 -2.635515,0 -0.547157,-0.547688 -0.682624,-1.352021 -0.405077,-2.026179 l -2.513541,-2.513277 v 6.613789 c 0.177271,0.08784 0.344753,0.204788 0.49239,0.352425 0.727604,0.727604 0.727604,1.906588 0,2.634986 -0.727604,0.727339 -1.907381,0.727339 -2.634456,0 -0.727604,-0.728663 -0.727604,-1.907646 0,-2.634986 0.179916,-0.179652 0.388144,-0.315648 0.610394,-0.406664 v -6.675438 c -0.22225,-0.09102 -0.429948,-0.225689 -0.610394,-0.406664 -0.551127,-0.550863 -0.683684,-1.359959 -0.401109,-2.036763 l -2.756164,-2.756694 -7.278688,7.277894 c -0.611452,0.611981 -0.611452,1.603375 0,2.214827 l 10.615348,10.614554 c 0.611188,0.611452 1.602317,0.611452 2.214298,0 L 88.304634,20.381888 c 0.61145,-0.611452 0.61145,-1.603375 0,-2.214827 z"
d="m 282.83,170.73 -0.27,-0.69 -26.14,-68.22 a 6.81,6.81 0 0 0 -2.69,-3.24 7,7 0 0 0 -8,0.43 7,7 0 0 0 -2.32,3.52 l -17.65,54 h -71.47 l -17.65,-54 a 6.86,6.86 0 0 0 -2.32,-3.53 7,7 0 0 0 -8,-0.43 6.87,6.87 0 0 0 -2.69,3.24 L 97.44,170 l -0.26,0.69 a 48.54,48.54 0 0 0 16.1,56.1 l 0.09,0.07 0.24,0.17 39.82,29.82 19.7,14.91 12,9.06 a 8.07,8.07 0 0 0 9.76,0 l 12,-9.06 19.7,-14.91 40.06,-30 0.1,-0.08 a 48.56,48.56 0 0 0 16.08,-56.04 z"
id="path348"/><path
class="cls-2"
d="m 282.83,170.73 -0.27,-0.69 a 88.3,88.3 0 0 0 -35.15,15.8 L 190,229.25 c 19.55,14.79 36.57,27.64 36.57,27.64 l 40.06,-30 0.1,-0.08 a 48.56,48.56 0 0 0 16.1,-56.08 z"
id="path350"/><path
class="cls-3"
d="m 153.43,256.89 19.7,14.91 12,9.06 a 8.07,8.07 0 0 0 9.76,0 l 12,-9.06 19.7,-14.91 c 0,0 -17.04,-12.89 -36.59,-27.64 -19.55,14.75 -36.57,27.64 -36.57,27.64 z"
id="path352"/><path
class="cls-2"
d="M 132.58,185.84 A 88.19,88.19 0 0 0 97.44,170 l -0.26,0.69 a 48.54,48.54 0 0 0 16.1,56.1 l 0.09,0.07 0.24,0.17 39.82,29.82 c 0,0 17,-12.85 36.57,-27.64 z"
d="M 27.79,115.217 1.54,69.749 a 11.499,11.499 0 0 1 0,-11.499 L 27.79,12.783 a 11.5,11.5 0 0 1 9.96,-5.75 h 52.5 a 11.5,11.5 0 0 1 9.959,5.75 l 26.25,45.467 a 11.499,11.499 0 0 1 0,11.5 l -26.25,45.466 a 11.5,11.5 0 0 1 -9.959,5.75 h -52.5 a 11.499,11.499 0 0 1 -9.96,-5.75 z"
fill="url(#a)"
id="path547"
style="fill:url(#a)"/></g><g
transform="translate(0,-7.034)"
id="g559"><defs
id="defs552"><path
d="M 27.791,115.217 1.541,69.749 a 11.499,11.499 0 0 1 0,-11.499 l 26.25,-45.467 a 11.499,11.499 0 0 1 9.959,-5.75 h 52.5 a 11.5,11.5 0 0 1 9.96,5.75 l 26.25,45.467 a 11.499,11.499 0 0 1 0,11.5 l -26.25,45.466 a 11.499,11.499 0 0 1 -9.96,5.75 h -52.5 a 11.499,11.499 0 0 1 -9.959,-5.75 z"
d="M 84.7,43.236 H 43.264 c -0.667,0 -1.212,0.546 -1.212,1.214 v 8.566 c 0,0.666 0.546,1.212 1.212,1.212 H 84.7 c 0.667,0 1.213,-0.546 1.213,-1.212 v -8.568 c 0,-0.666 -0.545,-1.213 -1.212,-1.213 m -6.416,7.976 a 2.484,2.484 0 0 1 -2.477,-2.48 2.475,2.475 0 0 1 2.477,-2.477 c 1.37,0 2.48,1.103 2.48,2.477 a 2.48,2.48 0 0 1 -2.48,2.48 m 6.415,8.491 -41.436,0.002 c -0.667,0 -1.212,0.546 -1.212,1.214 v 8.565 c 0,0.666 0.546,1.213 1.212,1.213 H 84.7 c 0.667,0 1.213,-0.547 1.213,-1.213 v -8.567 c 0,-0.666 -0.545,-1.214 -1.212,-1.214 m -6.416,7.976 a 2.483,2.483 0 0 1 -2.477,-2.48 2.475,2.475 0 0 1 2.477,-2.477 2.48,2.48 0 1 1 0,4.956"
This folder contains various examples like custom detectors, scripts, etc. Feel free to contribute!
### Generic Detector
An often requested feature for TruffleHog is a generic detector. By default, we do not support generic detection as it would result in lots of false positives. However, if you want to attempt detect generic secrets you can use a custom detector.
@@ -44,6 +44,18 @@ If you think that something should be included outside of these guidelines, plea
- Go 1.17+
- Make
### Adding New Token Formats to an Existing Scanner
In some instances, services will update their token format, requiring a new regex to properly detect secrets in addition to supporting the previous token format. Accomodating this can be done without adding a net-new detector. [We provide a `Versioner` interface](https://github.com/trufflesecurity/trufflehog/blob/e18cfd5e0af1469a9f05b8d5732bcc94c39da49c/pkg/detectors/detectors.go#L30) that can be implemented.
1. Create a copy of the package and append `_v2` to the package and file names. Ex: `<packagename>/` -> `<packagename>_v2`, `<packagename>.go` -> `<packagename>_v2.go`
Note: Be sure to update the tests to reference the new secret values in GSM, or the tests will fail.
2. Implement the `Versioner` interface. [GitHub example implementation.](/pkg/detectors/github_old/github_old.go#L22)
3. Proceed from step 3 of [Creating a new Secret Scanner](#creating-a-new-secret-scanner)
### Creating a new Secret Scanner
1. Identify the Secret Detector name from the [/proto/detectors.proto](/proto/detectors.proto) `DetectorType` enum.
@@ -62,15 +74,31 @@ If you think that something should be included outside of these guidelines, plea
1. Add the test secret to GCP Secrets. See [managing test secrets](#managing-test-secrets)
2. Update the pattern regex and keywords. Try iterating with [regex101.com](http://regex101.com/).
3. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not.
* Make sure you understand [verification indeterminacy](#verification-indeterminacy).
4. Update the tests with these test cases at minimum:
1. Found and verified (using a credential loaded from GCP Secrets)
2. Found and unverified
3. Not found
4. Any false positive cases that you come across
2. Found and unverified (determinately, i.e. the secret is invalid)
3. Found and unverified (indeterminately due to timeout)
4. Found and unverified (indeterminately due to an unexpected API response)
5. Not found
6. Any false positive cases that you come across
5. Create a merge request for review. CI tests must be passing.
## Addendum
### Verification indeterminacy
There are two types of reasons that secret verification can fail:
* The candidate secret is not actually a valid secret.
* Something went wrong in the process unrelated to the candidate secret, such as a transient network error or an unexpected API response.
In Trufflehog parlance, the first type of verification response is called _determinate_ and the second type is called _indeterminate_. Verification code should distinguish between the two by returning an error object in the result struct **only** for indeterminate failures. In general, a verifier should return an error (indicating an indeterminate failure) in all cases that haven't been explicitly identified as determinate failure states.
For example, consider a hypothetical authentication endpoint that returns `200 OK` for valid credentials and `403 Forbidden` for invalid credentials. The verifier for this endpoint could make an HTTP request and use the response status code to decide what to return:
* A `200` response would indicate that verification succeeded. (Or maybe any `2xx` response.)
* A `403` response would indicate that verification failed **determinately** and no error object should be returned.
* Any other response would indicate that verification failed **indeterminately** and an error object should be returned.
### Managing Test Secrets
Do not embed test credentials in the test code. Instead, use GCP Secrets Manager.
@@ -80,7 +108,7 @@ Do not embed test credentials in the test code. Instead, use GCP Secrets Manager
Note: `/tmp/s` is a valid path on Linux. You will need to change that for Windows or OSX, otherwise you will see an error. On Windows you will also need to install [WSL](https://docs.microsoft.com/en-us/windows/wsl/install).
+ [Creating a new Secret Scanner](#creating-a-new-secret-scanner)
+ [Creating a new Secret Detector](#creating-a-new-secret-detector)
+ [Testing the Detector](#testing-the-detector)
* [Addendum](#addendum)
+ [Using a test secret file](#using-a-test-secret-file)
+ [Adding Protos in Windows](#adding-protos-in-windows)
## Getting Started
@@ -40,34 +40,48 @@ If you think that something should be included outside of these guidelines, plea
- Go 1.17+
- Make
### Creating a new Secret Scanner
### Adding New Token Formats to an Existing Scanner
1. Identify the Secret Detector name from the [/proto/detectors.proto](/proto/detectors.proto) `DetectorType` enum. If necessary, run `make protos` when adding new ones.
In some instances, services will update their token format, requiring a new regex to properly detect secrets in addition to supporting the previous token format. Accomodating this can be done without adding a net-new detector. [We provide a `Versioner` interface](https://github.com/trufflesecurity/trufflehog/blob/e18cfd5e0af1469a9f05b8d5732bcc94c39da49c/pkg/detectors/detectors.go#L30) that can be implemented.
2. Generate the Secret Detector
1. Create a copy of the package and append `_v2` to the package and file names. Ex: `<packagename>/` -> `<packagename>_v2`, `<packagename>.go` -> `<packagename>_v2.go`
Note: Be sure to update the tests to reference the new secret values in GSM, or the tests will fail.
2. Implement the `Versioner` interface. [GitHub example implementation.](/pkg/detectors/github_old/github_old.go#L22)
3. Proceed from step 3 of [Creating a new Secret Scanner](#creating-a-new-secret-scanner)
### Creating a new Secret Detector
1. Add a new Secret Detector enum to the [`DetectorType` list here](/proto/detectors.proto).
2. Run `make protos` to update the `.pb` files.
3. Generate the Secret Detector
```bash
go run hack/generate/generate.go detector <DetectorType enum name>
example: go run hack/generate/generate.go detector SampleAPI
```
4. Add the Secret Detector to TruffleHog's Default Detectors
3. Complete the secret detector.
Add the secret scanner to the [`pkg/engine/defaults.go`](https://github.com/trufflesecurity/trufflehog/blob/main/pkg/engine/defaults.go) file like [`github.com/trufflesecurity/trufflehog/v3/pkg/detectors/<detector_name>`](https://github.com/trufflesecurity/trufflehog/blob/b71ea27a696bdf1c3141f637fda4ee4936c2f2d6/pkg/engine/defaults.go#L9) and
The previous step templated a boilerplate + some example code as a package in the `pkg/detectors` folder for you to work on.
The secret detector can be completed with these general steps:
The Secret Detector can be completed with these general steps:
1. Create a [test secrets file, and export the variable](#using-a-test-secret-file)
2. Update the pattern regex and keywords. Try iterating with [regex101.com](http://regex101.com/).
3. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not.
4. Update the tests with these test cases at minimum:
1. Found and verified (using a credential loaded from GCP Secrets)
2. Found and unverified
3. Not found
4. Any false positive cases that you come across
5. Create a pull request for review.
1. Update the pattern regex and keywords. Try iterating with [regex101.com](http://regex101.com/).
2. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not.
* Make sure you understand [verification indeterminacy](#verification-indeterminacy).
3. Create a [test for the detector](#testing-the-detector)
4. Create a pull request for review.
## Addendum
### Using a test secret file
### Testing the Detector
To ensure the quality of your PR, make sure your tests are passing with verified credentials.
1. Create a file called `.env` with this env file format:
@@ -81,8 +95,49 @@ If you think that something should be included outside of these guidelines, plea
```bash
export TEST_SECRET_FILE=".env"
```
The `.env` file should be in the new detector's directory like this:
```
├── tailscale
│ ├── .env
│ ├── tailscale.go
│ └── tailscale_test.go
```
Now that a `.env` file is present, the test file can load secrets locally.
Now, the detector test should attempt to load the given env key from that file.
3. Next, update the tests as necessary. A test file has already been generated by the `go run hack/generate/generate.go` command from earlier. There are 5 cases that have been generated:
1. Found and verified (using a credential loaded from the .env file)
2. Found and unverified (determinately, i.e. the secret is invalid)
3. Found and unverified (indeterminately due to timeout)
4. Found and unverified (indeterminately due to an unexpected API response)
5. Not found
Make any necessary updates to the tests. Note there might not be any changes required as the tests generated by the `go run hack/generate/generate.go` command are pretty good.
[Here is an exemplary test file for a detector which covers all 5 test cases](https://github.com/trufflesecurity/trufflehog/blob/6f9065b0aae981133a7fa3431c17a5c6213be226/pkg/detectors/browserstack/browserstack_test.go).
4. Now run the tests and check to make sure they are passing ✔️!
```bash
go test ./pkg/detectors/<detector> -tags=detectors
```
If the tests are passing, feel free to open a PR!
## Addendum
### Verification indeterminacy
There are two types of reasons that secret verification can fail:
* The candidate secret is not actually a valid secret.
* Something went wrong in the process unrelated to the candidate secret, such as a transient network error or an unexpected API response.
In Trufflehog parlance, the first type of verification response is called _determinate_ and the second type is called _indeterminate_. Verification code should distinguish between the two by returning an error object in the result struct **only** for indeterminate failures. In general, a verifier should return an error (indicating an indeterminate failure) in all cases that haven't been explicitly identified as determinate failure states.
For example, consider a hypothetical authentication endpoint that returns `200 OK` for valid credentials and `403 Forbidden` for invalid credentials. The verifier for this endpoint could make an HTTP request and use the response status code to decide what to return:
* A `200` response would indicate that verification succeeded. (Or maybe any `2xx` response.)
* A `403` response would indicate that verification failed **determinately** and no error object should be returned.
* Any other response would indicate that verification failed **indeterminately** and an error object should be returned.
### Adding Protos in Windows
@@ -100,3 +155,4 @@ Now, the detector test should attempt to load the given env key from that file.
archiveMaxSize=cli.Flag("archive-max-size","Maximum size of archive to scan. (Byte units eg. 512B, 2KB, 4MB)").Bytes()
archiveMaxDepth=cli.Flag("archive-max-depth","Maximum depth of archive to scan.").Int()
archiveTimeout=cli.Flag("archive-timeout","Maximum time to spend extracting an archive.").Duration()
includeDetectors=cli.Flag("include-detectors","Comma separated list of detector types to include. Protobuf name or IDs may be used, as well as ranges.").Default("all").String()
excludeDetectors=cli.Flag("exclude-detectors","Comma separated list of detector types to exclude. Protobuf name or IDs may be used, as well as ranges. IDs defined here take precedence over the include list.").String()
gitScan=cli.Command("git","Find credentials in git repositories.")
gitScanURI=gitScan.Arg("uri","Git repository URL. https:// or file:// schema expected.").Required().String()
gitScanURI=gitScan.Arg("uri","Git repository URL. https://, file://, or ssh:// schema expected.").Required().String()
gitScanIncludePaths=gitScan.Flag("include-paths","Path to file with newline separated regexes for files to include in scan.").Short('i').String()
gitScanExcludePaths=gitScan.Flag("exclude-paths","Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
gitScanExcludeGlobs=gitScan.Flag("exclude-globs","Comma separated list of globs to exclude in scan. This option filters at the `git log` level, resulting in faster scans.").String()
gitScanSinceCommit=gitScan.Flag("since-commit","Commit to start scan from.").String()
gitScanBranch=gitScan.Flag("branch","Branch to scan.").String()
gitScanMaxDepth=gitScan.Flag("max-depth","Maximum depth of commits to scan.").Int()
gitScanBare=gitScan.Flag("bare","Scan bare repository (e.g. useful while using in pre-receive hooks)").Bool()
_=gitScan.Flag("allow","No-op flag for backwards compat.").Bool()
_=gitScan.Flag("entropy","No-op flag for backwards compat.").Bool()
_=gitScan.Flag("regex","No-op flag for backwards compat.").Bool()
githubScan=cli.Command("github","Find credentials in GitHub repositories.")
githubScanRepos=githubScan.Flag("repo",`GitHub repository to scan. You can repeat this flag. Example: "https://github.com/dustin-decker/secretsandstuff"`).Strings()
githubScanOrgs=githubScan.Flag("org",`GitHub organization to scan. You can repeat this flag. Example: "trufflesecurity"`).Strings()
githubScanRepos=githubScan.Flag("repo",`GitHub repository to scan. You can repeat this flag. Example: "https://github.com/dustin-decker/secretsandstuff"`).Strings()
githubScanOrgs=githubScan.Flag("org",`GitHub organization to scan. You can repeat this flag. Example: "trufflesecurity"`).Strings()
githubScanToken=githubScan.Flag("token","GitHub token. Can be provided with environment variable GITHUB_TOKEN.").Envar("GITHUB_TOKEN").String()
githubIncludeForks=githubScan.Flag("include-forks","Include forks in scan.").Bool()
githubIncludeMembers=githubScan.Flag("include-members","Include organization member repositories in scan.").Bool()
githubIncludeRepos=githubScan.Flag("include-repos",`Repositories to include in an org scan. This can also be a glob pattern. You can repeat this flag. Must use Github repo full name. Example: "trufflesecurity/trufflehog", "trufflesecurity/t*"`).Strings()
githubExcludeRepos=githubScan.Flag("exclude-repos",`Repositories to exclude in an org scan. This can also be a glob pattern. You can repeat this flag. Must use Github repo full name. Example: "trufflesecurity/driftwood", "trufflesecurity/d*"`).Strings()
githubScanIncludePaths=githubScan.Flag("include-paths","Path to file with newline separated regexes for files to include in scan.").Short('i').String()
githubScanExcludePaths=githubScan.Flag("exclude-paths","Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
githubScanIssueComments=githubScan.Flag("issue-comments","Include issue descriptions and comments in scan.").Bool()
githubScanPRComments=githubScan.Flag("pr-comments","Include pull request descriptions and comments in scan.").Bool()
githubScanGistComments=githubScan.Flag("gist-comments","Include gist comments in scan.").Bool()
gitlabScan=cli.Command("gitlab","Find credentials in GitLab repositories.")
gitlabScanRepos=gitlabScan.Flag("repo","GitLab repo url. You can repeat this flag. Leave empty to scan all repos accessible with provided credential. Example: https://gitlab.com/org/repo.git").Strings()
gitlabScanRepos=gitlabScan.Flag("repo","GitLab repo url. You can repeat this flag. Leave empty to scan all repos accessible with provided credential. Example: https://gitlab.com/org/repo.git").Strings()
gitlabScanToken=gitlabScan.Flag("token","GitLab token. Can be provided with environment variable GITLAB_TOKEN.").Envar("GITLAB_TOKEN").Required().String()
gitlabScanIncludePaths=gitlabScan.Flag("include-paths","Path to file with newline separated regexes for files to include in scan.").Short('i').String()
gitlabScanExcludePaths=gitlabScan.Flag("exclude-paths","Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
filesystemScan=cli.Command("filesystem","Find credentials in a filesystem.")
filesystemDirectories=filesystemScan.Flag("directory","Path to directory to scan. You can repeat this flag.").Required().Strings()
filesystemScan=cli.Command("filesystem","Find credentials in a filesystem.")
filesystemPaths=filesystemScan.Arg("path","Path to file or directory to scan.").Strings()
// DEPRECATED: --directory is deprecated in favor of arguments.
filesystemDirectories=filesystemScan.Flag("directory","Path to directory to scan. You can repeat this flag.").Strings()
// TODO: Add more filesystem scan options. Currently only supports scanning a list of directories.
// filesystemScanIncludePaths = filesystemScan.Flag("include-paths", "Path to file with newline separated regexes for files to include in scan.").Short('i').String()
// filesystemScanExcludePaths = filesystemScan.Flag("exclude-paths", "Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
filesystemScanIncludePaths=filesystemScan.Flag("include-paths","Path to file with newline separated regexes for files to include in scan.").Short('i').String()
filesystemScanExcludePaths=filesystemScan.Flag("exclude-paths","Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
s3Scan=cli.Command("s3","Find credentials in S3 buckets.")
s3ScanKey=s3Scan.Flag("key","S3 key used to authenticate.").String()
s3ScanSecret=s3Scan.Flag("secret","S3 secret used to authenticate.").String()
s3ScanCloudEnv=s3Scan.Flag("cloud-environment","Use IAM credentials in cloud environment.").Bool()
s3ScanBuckets=s3Scan.Flag("bucket","Name of S3 bucket to scan. You can repeat this flag.").Strings()
s3Scan=cli.Command("s3","Find credentials in S3 buckets.")
s3ScanKey=s3Scan.Flag("key","S3 key used to authenticate. Can be provided with environment variable AWS_ACCESS_KEY_ID.").Envar("AWS_ACCESS_KEY_ID").String()
s3ScanRoleArns=s3Scan.Flag("role-arn","Specify the ARN of an IAM role to assume for scanning. You can repeat this flag.").Strings()
s3ScanSecret=s3Scan.Flag("secret","S3 secret used to authenticate. Can be provided with environment variable AWS_SECRET_ACCESS_KEY.").Envar("AWS_SECRET_ACCESS_KEY").String()
s3ScanSessionToken=s3Scan.Flag("session-token","S3 session token used to authenticate temporary credentials. Can be provided with environment variable AWS_SESSION_TOKEN.").Envar("AWS_SESSION_TOKEN").String()
s3ScanCloudEnv=s3Scan.Flag("cloud-environment","Use IAM credentials in cloud environment.").Bool()
s3ScanBuckets=s3Scan.Flag("bucket","Name of S3 bucket to scan. You can repeat this flag.").Strings()
s3ScanMaxObjectSize=s3Scan.Flag("max-object-size","Maximum size of objects to scan. Objects larger than this will be skipped. (Byte units eg. 512B, 2KB, 4MB)").Default("250MB").Bytes()
gcsScan=cli.Command("gcs","Find credentials in GCS buckets.")
gcsProjectID=gcsScan.Flag("project-id","GCS project ID used to authenticate. Can NOT be used with unauth scan. Can be provided with environment variable GOOGLE_CLOUD_PROJECT.").Envar("GOOGLE_CLOUD_PROJECT").String()
gcsCloudEnv=gcsScan.Flag("cloud-environment","Use Application Default Credentials, IAM credentials to authenticate.").Bool()
gcsServiceAccount=gcsScan.Flag("service-account","Path to GCS service account JSON file.").ExistingFile()
gcsWithoutAuth=gcsScan.Flag("without-auth","Scan GCS buckets without authentication. This will only work for public buckets").Bool()
gcsAPIKey=gcsScan.Flag("api-key","GCS API key used to authenticate. Can be provided with environment variable GOOGLE_API_KEY.").Envar("GOOGLE_API_KEY").String()
gcsIncludeBuckets=gcsScan.Flag("include-buckets","Buckets to scan. Comma separated list of buckets. You can repeat this flag. Globs are supported").Short('I').Strings()
gcsExcludeBuckets=gcsScan.Flag("exclude-buckets","Buckets to exclude from scan. Comma separated list of buckets. Globs are supported").Short('X').Strings()
gcsIncludeObjects=gcsScan.Flag("include-objects","Objects to scan. Comma separated list of objects. you can repeat this flag. Globs are supported").Short('i').Strings()
gcsExcludeObjects=gcsScan.Flag("exclude-objects","Objects to exclude from scan. Comma separated list of objects. You can repeat this flag. Globs are supported").Short('x').Strings()
gcsMaxObjectSize=gcsScan.Flag("max-object-size","Maximum size of objects to scan. Objects larger than this will be skipped. (Byte units eg. 512B, 2KB, 4MB)").Default("10MB").Bytes()
syslogScan=cli.Command("syslog","Scan syslog")
syslogAddress=syslogScan.Flag("address","Address and port to listen on for syslog. Example: 127.0.0.1:514").String()
@@ -87,6 +133,12 @@ var (
syslogTLSCert=syslogScan.Flag("cert","Path to TLS cert.").String()
syslogTLSKey=syslogScan.Flag("key","Path to TLS key.").String()
syslogFormat=syslogScan.Flag("format","Log format. Can be rfc3164 or rfc5424").String()
circleCiScanToken=circleCiScan.Flag("token","CircleCI token. Can also be provided with environment variable").Envar("CIRCLECI_TOKEN").Required().String()
dockerScanImages=dockerScan.Flag("image","Docker image to scan. Use the file:// prefix to point to a local tarball, otherwise a image registry is assumed.").Required().Strings()
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Miccah Castorina
