* add sharepoint source proto
* create sharepoint oss protos
* add email field, remove oauth2 type, update token to refresh_token
* rename path to link
* restore clientcredentials
* restore s3 and confluence proto changes from make command
* Add in-memory caching lib, used by the GCS source.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* Fix static check.
* Add test for NewWithData.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* update comment.
* update comments.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* remove unused dep.
* address comments.
* Add exists method.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* fix test.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* rebase.
* split encode resume by comma.
* Use a persistable cache.
* fix merge.
* fix merge.
* Add progress as part of the cache given it will be the persistence layer.
* Add test for making sure the cache doesn't persist when the increment value is not met.
* fix tests.
* Resolve#1167 by adding support for the AWS_SESSION_TOKEN environment variable and adding a --session-token cli arg
* fix error message
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
* Adjust types to use DetectorID struct
* Parse versions with detector include and exclude input
* Update detectors filter to use version
Co-authored-by: steeeve <steve@trufflesec.com>
* Implement Versioner for github, gitlab, and npm detectors
Co-authored-by: steeeve <steve@trufflesec.com>
---------
Co-authored-by: steeeve <steve@trufflesec.com>
* Refactor generating CustomRegex results into a helper function
* Added errGroup for createResults, and ensure goroutines are non-blocking
* clean return
---------
Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>
This will concatenate all errors together into a single string. When
possible, it would be better to log the actual errors slice to take
advantage of structured logging.
* Add ability to include and exclude detectors
* Trim space before checking for empty items
* Explicitly check for integer overflow
* Use strconv.ParseInt instead of strconv.Atoi
* Address comments
* Rename directories to paths
* Generate protos
* Add file scanning support to filesystem source
* Add directories back to filesystem proto
* Generate protos
* Combine paths and directories from in source
* Add filesystem filter
* Address comments
In this case just Heroku and LinearAPI. But this includes the Moonclerck
detector, which appears to be a typo that got turned into a separate
detector type.
Co-authored-by: zubairk14 <zubair.khan@trufflesec.com>
* Add Type() to detector interface
The goal here is to allow the detector type information to be used
without the need for reflection. This could possibly allow us to more
easily inject information into detectors or filter them out if
necessary.
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
* remove test detector
---------
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
Checking for false positives can lead to results being removed before
ever getting the opportunity to verify them. Users are already
responsible for verification of custom detectors, so let's not interfere
with how they choose to use it.
* Adding missing flags to Readme
* Use retryableHttpClient by default for GitHub
* Adding repoUrl for scanning time log
* Use WithField instead of WithFields
* Updating README with lasted --help output
* Allow using a glob for include list.
* Update command flag.
* Make comment more clear.
* update comment.
* Allow scanning repo and org at the same time.
* Remove verifying successRanges because it is unused in webhook
* Move custom_detectors validation code into its own file
* Initial implementation of custom regex detector
Secret verification is done via webhook.
* Add CustomRegex detector type
* Add upper bound to permutation
* Return early if the context is canceled
* Add headers from configuration
* Add detector name as a key in the JSON body
* Implement faster algorithm for productIndices
* Add validation skeleton
* Add custom detector validation with tests
* Validate and test regex vars
* Implement RegexVarString
* Use RegexVarString for validating regex variables
* Add numerics to the regex variable matching
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add custom_detectors proto
* Generate proto code
* Create custom_detectors package
Also create protoyaml package to test YAML unmarshalling the
configuration.
* Simplify custom_detectors proto by removing connection
* Generate proto code
* Update custom_detectors parsing tests
Fixing help subcommand reference to be `--help`, not `-h`, which does not work:
```
$ trufflehog git -h
trufflehog: error: unknown short flag '-h', try --help
$ trufflehog --version
trufflehog 3.16.4
```
* Fixed the brew install tap command
It seems you need to run the full tap, not just trufflehog.
Signed-off-by: JJ Asghar <awesome@ibm.com>
* Update README.md
Signed-off-by: JJ Asghar <awesome@ibm.com>
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
* simplify monogo pattern
* do URI verification locally, while attempting to defuse SSRF
* test SSRF defuse
* simplify err check logic per linter recommendation
* split up detectors
* address comments
* remove unused var
* Remove the check to filter and return only a single unverified result.
* Revert "Remove the check to filter and return only a single unverified result."
This reverts commit 494e432803.
* Add new CLI flag to filter unverified results.
Also, don't fetch a github user or their token when both are known. This
currently only affects the Github Token auth type. Github App
installations will continually fetch tokens every time we clone a repo.
In the future we should check the `ExpiresAt` field of the Github App
token and determine if we need to fetch a new one at that point.
* Add SSH config option for the git source
The auth message is empty since we use the git binary underneath to
handle the SSH authentication.
* Import digitaloceanv2
I'm not sure I fully understand why this issue exists. But I think the
short version is this: When we attempted to paginate users, we would set
a variable's Page value. But that variable appears to not actually be a
pointer, despite being added as one. It probably has to do with how
struct embedding works. Either way, if we make the overall options
variable the whole thing, and update its embedded struct with our page
variable, everything works out.
* [THOG-162] Implement JDBC verification for select drivers
Also includes integration tests for postgres and mysql via docker. To
run, execute the following (untested what will happen if the docker
images aren't installed):
go test -tags=detectors,integration ./pkg/detectors/jdbc
* Make jdbc regex a bit more strict
* Surface the context to allow the caller to set a timeout
* Update readme.go
Readme has change the way they issue their keys
now its like rdme_{70} ascii chars
* Update readme.go
* Update readme.go
* Update readme.go
The tester seems to be working fine with the new defaultclient code
* Handle errors w/ github source.
* Fix loop var captured by func literal.
* Fix loop var captured by func literal.
* Set completed progress if the scan completes with no errors.
* Set progress to 100% if the scope and iteration are both 0.
* Fix commentary.
* Fix test.
* Return after the defer to os.RemoveAll.
* Fix unauth scan.
* Inline range loop.
* update tests for partial scan completion with errors. Ensure correct progress is set.
* Update progress for all sources.
* Update github test.
* Address comments.
* [THOG-643] Implement independent log level controls
There are two log level controls to mentally distinguish. Log levels
associated with a sink (e.g. stdout and streamed), and log levels
associated with a logger (e.g. a GitHub source).
The level is determined to be the minimum of the two. If a sink is at
level 0, then it will only output level 0 logs regardless of the
logger's level. This is best demonstrated by TestSinkWithName.
* Rename WithName to WithNamedLevel
* Check flush errors
* Replace IncreaseLevelCore with custom LevelCore
Adding a leveler that was less verbose would cause the initialization
fail, and therefore not be added to the core. This check is only at
the time of initialization.
An alternative approach to creating our own core is to set the child log
level equal to the parent, so initialization is guaranteed (with the
added benefit of intuitive behavior).
* Use controller if it exists, otherwise inherit parent's log level
* Cleanup some tests
* Fix the starting index value for plus line check.
* Set the correct source type for notifications.
* Reset old value.
* Fix the starting index value for plus line check.
* Fix len check.
* Reset old value.
* Add tests.
* Update tests.
* Update tests.
* Start updating detectors that have two part creds to record the raw result as ID + secret.
* Add more detectors.
* More detectors.
* More detectors.
* remove comment out imports.
* Scan binary files for git sources
* Create data chunks in for loop
* Linter feedback and newline commit result
* Use disk buffered reader and chunker function
* Use a config struct when scanning and engine source.
* fix tests.
* Move test_helpers to the sources pkg.
* Handle ScanGit error in tests.
* adderss comments.
* Use functional options.
* Remove temp var.
* Add better var names for the setup functions for each config.
* Remove unused var.
* fix error logs.
* fix error logs.
* single line.
* remove blank lines.
The fragment trace was a bit too verbose even at the trace level. We may
want to trace the file being chunked or something like that, but not the
entire diff.
* added common regex patterns for detectors
* For HexPattern
* enhancements
* used parseInt
* enhancement
* enhanced regex for email and subdomain
* enhancement for email pattern
* update pattern and detector
Co-authored-by: Roxanne Tampus <roxannetampus02@gmail.com>
* Remove the ending word boundary for the AWS dectector. This will prevent missing secrets that end with / due to it not being ASCII.
* Update regex to be more strict.
* Refactor Engine to wait for workers in a Finish method
This should allow the engine to run multiple concurrent scans if
desired before shutting down.
Additionally, this commit refactors some of the printing logic to the
output package.
* Fix tests
* Shallow clone if --since-commit is provided
* Set the user before constructing args
* Fix vbout detector
* Address PR comments
* Use a better name for timestamp
* Use net.URL.String method for the remote path
* Use the string version of the Detector Type.
* Only modify the output for json.
* reorder import.
* Fix imports.
* Add DetectorName in addition to DetectorType to the json output.
The 'runs.image:' argument was using `Dockerfile`. This caused
a Docker build of the trufflehog image every time someone
used this action.
I've changed the argument to use the already published Docker image.
This will speed up the workflows considerably, since it pulls the
image that is already built and available on GitHub's image storage.
* Add unit tests and refactor some logic
* Move integration tests to a separate file behind a build flag
* Fix bugs in normalizeRepos
* Address lint errors
* Sort slices before comparing because order doesn't matter
* Reorganize GitHub source
This breaks up the Chunks method into smaller sub-method calls to help
organize and better understand the logic flow. No logic has been
modified (except one obvious bug), just shuffling code around.
* Check errors and revert bug fix
* fix: regex for ends with github.com
* fix: regex for hooks.zapier.com
* fix: regex for hooks.slack.com
* fix: regex for signalwire.com
* fix: regex for kanbantool.com
* fix: regex for invoiceoceam.com
* fix: regex for invoiceocean.com
* fix: regex for freshdesk.com
* fix: regex for discord.com
* fix: regex for deputy.com
* Fix broken gitlab test.
* Close chunks chan from blocking indefinitely.
* Range over chunksCh in the event chunksCh is nil we don't run into an invalid memory address error.
Update warnings and move clone output information back.
* Remove commented out code.
* Remove .Run() because .CombinedOutput() should call .Run()
* Update test to include count check.
* Address PR comments.
* Fix merge issue.
* Use oauth2 as the username for cloning private repos with a token.
* Update username string.
* Only set user to "placeholder" if no username is present.
* Detectors that fail verification should still report the unverified secret
* fixup - change microsoft webhook keywords, filter false positives for old github detector
* fixup - fix typo
* Log explanation of non-zero exit with found results
* Change exit code and add documentation
* Adjust exit code handling
* Make action fail on found results
* Use new overseer
* Improve wording
* Update README.md
* Add RechargePayments to detectors
* First pass at code and tests for RechargePayments detector
* Running make protos
* Fixes based on running tests
Co-authored-by: Kevin Stilwell <kevin.stilwell@gmail.com>
* Update detectors PrefixRegex to allow for new line and carriage returns.
Add additional keyword for security trails.
Add additional unit tests for security trails and PrefixRegex
* Update catpure group.
A clear and concise description of what the bug is.
### Community Note
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Expected behavior**
A clear and concise description of what you expected to happen.
### TruffleHog Version
<!--- Please run `trufflehog --version` to show the version. If you are not running the latest version, please upgrade because your issue may have already been fixed. --->
**Output**
Please include relevant log output, and run with the `--debug` flag.
### Trace Output
**Screenshots**
If applicable, add screenshots to help explain your problem.
<!---
Please provide a link to a GitHub Gist containing the complete debug output. Please do NOT paste the debug output in the issue; just paste a link to the Gist.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Version [e.g. 22]
To obtain the trace output, run trufflehog with the --trace flag.
--->
### Expected Behavior
<!--- What should have happened? --->
### Actual Behavior
<!--- What actually happened? --->
### Steps to Reproduce
<!--- Please list the steps required to reproduce the issue. --->
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
## Environment
* OS: [e.g. iOS]
* Version [e.g. 22]
## Additional Context
<!--- Add any other context about the problem here. --->
### References
<!---
Information about referencing Github Issues: https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests
Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Vendor documentation? For example:
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
### Community Note
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
### Description
**Additional context**
Add any other context or screenshots about the feature request here.
<!--- Please leave a helpful description of the feature request here. --->
## Problem to be Addressed
<!--- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] --->
## Description of the Preferred Solution
<!--- A clear and concise description of what you want to happen. What
information may be required and what would be the preferred way to provide it?
What should the output include? --->
## Additional Context
<!--- Add any other context or screenshots about the feature request here. --->
### References
<!---
Information about referencing Github Issues: https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests
Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Vendor blog posts or documentation? For example:
Most testing is handled automatically by our GitHub Actions workflows.
## Local GitHub Action Testing
In some cases you may wish to submit changes to the Trufflehog GitHub Action. Unfortunately GitHub does not provide a 1st-party testing environment for testing actions outside of GitHub Actions.
Fortunately [nektos/act](https://github.com/nektos/act) enables local testing of GitHub Actions.
### Instructions
1. Please follow [the installation instructions](http://https://github.com/nektos/act#installation) for your OS.
2. The first run of `act` will ask you to specify an image. `Medium` should suffice.
3. You'll need to configure a personal-access-token(PAT) with: `repo:status`, `repo_deployment`, and `public_repo` permissions.
4. Set an environment variable named `GITHUB_TOKEN` with the PAT from the previous step as the value: `$ export GITHUB_TOKEN=<your_PAT>`
5. Run the following command from the repository root: `act pull_request -j test -W .github/workflows/secrets.yml -s GITHUB_TOKEN --defaultbranch main`
6. If the job was successful, you should expect to see output from the scanner showing several detected secrets.
7. If you want to omit the context of a pull request event and just test that the action starts successfully, run: `act -j test -W .github/workflows/secrets.yml -s GITHUB_TOKEN --defaultbranch main`
For every potential credential that is detected, we've painstakingly implemented programatic verification against the API that we think it belongs to. Verification eliminates false positives. For example, the [AWS credential detector](pkg/detectors/aws/aws.go) performs a `GetCallerIdentity` API call against the AWS API to verify if an AWS credential is active.
# Binary releases
Download and unpack from https://github.com/trufflesecurity/trufflehog/releases
+ All I see is `🐷🔑🐷 TruffleHog. Unearth your secrets. 🐷🔑🐷` and the program exits, what gives?
+ That means no secrets were detected
+ Why is the scan is taking a long time when I scan a GitHub org
+ Unauthenticated GitHub scans have rate limits. To improve your rate limits, include the `--token` flag with a personal access token
+ It says a private key was verified, what does that mean?
+ Check out our Driftwood blog post to learn how to do this, in short we've confirmed the key can be used live for SSH or SSL [Blog post](https://trufflesecurity.com/blog/driftwood-know-if-private-keys-are-sensitive/)
# :newspaper: What's new in v3?
TruffleHog v3 is a complete rewrite in Go with many new powerful features.
- We've **added over 700 credential detectors that support active verification against their respective APIs**.
- We've also added native **support for scanning GitHub, GitLab, filesystems, S3, GCS and Circle CI**.
- **Instantly verify private keys** against millions of github users and **billions** of TLS certificates using our [Driftwood](https://trufflesecurity.com/blog/driftwood) technology.
- Scan binaries and other file formats
- Available as a GitHub Action and a pre-commit hook
## What is credential verification?
For every potential credential that is detected, we've painstakingly implemented programmatic verification against the API that we think it belongs to. Verification eliminates false positives. For example, the [AWS credential detector](pkg/detectors/aws/aws.go) performs a `GetCallerIdentity` API call against the AWS API to verify if an AWS credential is active.
# :memo: Usage
TruffleHog has a sub-command for each source of data that you may want to scan:
@@ -94,10 +165,13 @@ TruffleHog has a sub-command for each source of data that you may want to scan:
- github
- gitlab
- S3
- filesystem
-file and stdin (coming soon)
- filesystem (files and directories)
-syslog
- circleci
- GCS (Google Cloud Storage)
- stdin (coming soon)
Each subcommand can have options that you can see with the `-h` flag provided to the sub command:
Each subcommand can have options that you can see with the `--help` flag provided to the sub command:
```
$ trufflehog git --help
@@ -107,29 +181,23 @@ Find credentials in git repositories.
Flags:
--help Show context-sensitive help (also try --help-long and --help-man).
--debug Run in debug mode
--versionPrints trufflehog version.
--debug Run in debug mode.
--trace Run in trace mode.
--profile Enables profiling and sets a pprof and fgprof server on :18066.
-j, --json Output in JSON format.
--json-legacy Use the pre-v3.0 JSON format. Only works with git, gitlab, and github sources.
--concurrency=1 Number of concurrent workers.
--concurrency=10 Number of concurrent workers.
--no-verification Don't verify the results.
--only-verified Only output verified results.
--filter-unverified Only output first unverified result per chunk per detector if there are more than one results.
--config=CONFIG Path to configuration file.
--print-avg-detector-time Print the average time spent on each detector.
--no-update Don't check for updates.
-i, --include-paths=INCLUDE-PATHS
Path to file with newline separated regexes for files to include in scan.
-x, --exclude-paths=EXCLUDE-PATHS
Path to file with newline separated regexes for files to exclude in scan.
--since-commit=SINCE-COMMIT
Commit to start scan from.
--branch=BRANCH Branch to scan.
--max-depth=MAX-DEPTH Maximum depth of commits to scan.
--allow No-op flag for backwards compat.
--entropy No-op flag for backwards compat.
--regex No-op flag for backwards compat.
--fail Exit with code 183 if results are found.
--versionShow application version.
Args:
<uri> Git repository URL. https:// or file:// schema expected.
<uri> Git repository URL. https://, file://, or ssh:// schema expected.
```
For example, to scan a `git` repository, start with
@@ -138,17 +206,164 @@ For example, to scan a `git` repository, start with
This project exists thanks to all the people who contribute. [[Contribute](CONTRIBUTING.md)].
@@ -158,16 +373,29 @@ This project exists thanks to all the people who contribute. [[Contribute](CONTR
</a>
## Contributing
# :computer: Contributing
Contributions are very welcome! Please see our [contribution guidelines first](CONTRIBUTING.md).
We no longer accept contributions to TruffleHog v2, but that code is available in the `v2` branch.
### Adding new secret detectors
## Adding new secret detectors
We have published some [documentation and tooling to get started on adding new secret detectors](hack/docs/Adding_Detectors_external.md). Let's improve detection together!
## License Change
# Use as a library
Currently, trufflehog is in heavy development and no guarantees can be made on
the stability of the public APIs at this time.
# License Change
Since v3.0, TruffleHog is released under a AGPL 3 license, included in [`LICENSE`](LICENSE). TruffleHog v3.0 uses none of the previous codebase, but care was taken to preserve backwards compatibility on the command line interface. The work previous to this release is still available licensed under GPL 2.0 in the history of this repository and the previous package releases and tags. A completed CLA is required for us to accept contributions going forward.
# :money_with_wings: Enterprise product
Are you interested in continously monitoring your Git, Jira, Slack, Confluence, etc.. for credentials? We have an enterprise product that can help. Reach out here to learn more https://trufflesecurity.com/contact/
We take the revenue from the enterprise product to fund more awesome open source projects that the whole community can benefit from.
d="M 27.791,115.217 1.541,69.749 a 11.499,11.499 0 0 1 0,-11.499 l 26.25,-45.467 a 11.499,11.499 0 0 1 9.959,-5.75 h 52.5 a 11.5,11.5 0 0 1 9.96,5.75 l 26.25,45.467 a 11.499,11.499 0 0 1 0,11.5 l -26.25,45.466 a 11.499,11.499 0 0 1 -9.96,5.75 h -52.5 a 11.499,11.499 0 0 1 -9.959,-5.75 z"
id="b"/></defs><g
inkscape:label="Layer 1"
inkscape:groupmode="layer"
id="layer1"
transform="translate(-5.467857,-5.738096)"><g
id="g706"><path
fill="#f05133"
d="M 88.304634,18.167061 77.689818,7.5527709 c -0.611187,-0.611452 -1.602581,-0.611452 -2.214298,0 l -2.203979,2.204508 2.795852,2.7958521 c 0.649817,-0.219604 1.394883,-0.07223 1.912673,0.445823 0.520965,0.521229 0.667015,1.272646 0.441854,1.924844 l 2.695045,2.694781 c 0.652198,-0.224895 1.404144,-0.07938 1.924844,0.442119 0.727604,0.727604 0.727604,1.906588 0,2.634721 -0.728134,0.727869 -1.907117,0.727869 -2.635515,0 -0.547157,-0.547688 -0.682624,-1.352021 -0.405077,-2.026179 l -2.513541,-2.513277 v 6.613789 c 0.177271,0.08784 0.344753,0.204788 0.49239,0.352425 0.727604,0.727604 0.727604,1.906588 0,2.634986 -0.727604,0.727339 -1.907381,0.727339 -2.634456,0 -0.727604,-0.728663 -0.727604,-1.907646 0,-2.634986 0.179916,-0.179652 0.388144,-0.315648 0.610394,-0.406664 v -6.675438 c -0.22225,-0.09102 -0.429948,-0.225689 -0.610394,-0.406664 -0.551127,-0.550863 -0.683684,-1.359959 -0.401109,-2.036763 l -2.756164,-2.756694 -7.278688,7.277894 c -0.611452,0.611981 -0.611452,1.603375 0,2.214827 l 10.615348,10.614554 c 0.611188,0.611452 1.602317,0.611452 2.214298,0 L 88.304634,20.381888 c 0.61145,-0.611452 0.61145,-1.603375 0,-2.214827 z"
d="m 282.83,170.73 -0.27,-0.69 -26.14,-68.22 a 6.81,6.81 0 0 0 -2.69,-3.24 7,7 0 0 0 -8,0.43 7,7 0 0 0 -2.32,3.52 l -17.65,54 h -71.47 l -17.65,-54 a 6.86,6.86 0 0 0 -2.32,-3.53 7,7 0 0 0 -8,-0.43 6.87,6.87 0 0 0 -2.69,3.24 L 97.44,170 l -0.26,0.69 a 48.54,48.54 0 0 0 16.1,56.1 l 0.09,0.07 0.24,0.17 39.82,29.82 19.7,14.91 12,9.06 a 8.07,8.07 0 0 0 9.76,0 l 12,-9.06 19.7,-14.91 40.06,-30 0.1,-0.08 a 48.56,48.56 0 0 0 16.08,-56.04 z"
id="path348"/><path
class="cls-2"
d="m 282.83,170.73 -0.27,-0.69 a 88.3,88.3 0 0 0 -35.15,15.8 L 190,229.25 c 19.55,14.79 36.57,27.64 36.57,27.64 l 40.06,-30 0.1,-0.08 a 48.56,48.56 0 0 0 16.1,-56.08 z"
id="path350"/><path
class="cls-3"
d="m 153.43,256.89 19.7,14.91 12,9.06 a 8.07,8.07 0 0 0 9.76,0 l 12,-9.06 19.7,-14.91 c 0,0 -17.04,-12.89 -36.59,-27.64 -19.55,14.75 -36.57,27.64 -36.57,27.64 z"
id="path352"/><path
class="cls-2"
d="M 132.58,185.84 A 88.19,88.19 0 0 0 97.44,170 l -0.26,0.69 a 48.54,48.54 0 0 0 16.1,56.1 l 0.09,0.07 0.24,0.17 39.82,29.82 c 0,0 17,-12.85 36.57,-27.64 z"
d="M 27.79,115.217 1.54,69.749 a 11.499,11.499 0 0 1 0,-11.499 L 27.79,12.783 a 11.5,11.5 0 0 1 9.96,-5.75 h 52.5 a 11.5,11.5 0 0 1 9.959,5.75 l 26.25,45.467 a 11.499,11.499 0 0 1 0,11.5 l -26.25,45.466 a 11.5,11.5 0 0 1 -9.959,5.75 h -52.5 a 11.499,11.499 0 0 1 -9.96,-5.75 z"
fill="url(#a)"
id="path547"
style="fill:url(#a)"/></g><g
transform="translate(0,-7.034)"
id="g559"><defs
id="defs552"><path
d="M 27.791,115.217 1.541,69.749 a 11.499,11.499 0 0 1 0,-11.499 l 26.25,-45.467 a 11.499,11.499 0 0 1 9.959,-5.75 h 52.5 a 11.5,11.5 0 0 1 9.96,5.75 l 26.25,45.467 a 11.499,11.499 0 0 1 0,11.5 l -26.25,45.466 a 11.499,11.499 0 0 1 -9.96,5.75 h -52.5 a 11.499,11.499 0 0 1 -9.959,-5.75 z"
d="M 84.7,43.236 H 43.264 c -0.667,0 -1.212,0.546 -1.212,1.214 v 8.566 c 0,0.666 0.546,1.212 1.212,1.212 H 84.7 c 0.667,0 1.213,-0.546 1.213,-1.212 v -8.568 c 0,-0.666 -0.545,-1.213 -1.212,-1.213 m -6.416,7.976 a 2.484,2.484 0 0 1 -2.477,-2.48 2.475,2.475 0 0 1 2.477,-2.477 c 1.37,0 2.48,1.103 2.48,2.477 a 2.48,2.48 0 0 1 -2.48,2.48 m 6.415,8.491 -41.436,0.002 c -0.667,0 -1.212,0.546 -1.212,1.214 v 8.565 c 0,0.666 0.546,1.213 1.212,1.213 H 84.7 c 0.667,0 1.213,-0.547 1.213,-1.213 v -8.567 c 0,-0.666 -0.545,-1.214 -1.212,-1.214 m -6.416,7.976 a 2.483,2.483 0 0 1 -2.477,-2.48 2.475,2.475 0 0 1 2.477,-2.477 2.48,2.48 0 1 1 0,4.956"
archiveMaxSize=cli.Flag("archive-max-size","Maximum size of archive to scan. (Byte units eg. 512B, 2KB, 4MB)").Bytes()
archiveMaxDepth=cli.Flag("archive-max-depth","Maximum depth of archive to scan.").Int()
archiveTimeout=cli.Flag("archive-timeout","Maximum time to spend extracting an archive.").Duration()
includeDetectors=cli.Flag("include-detectors","Comma separated list of detector types to include. Protobuf name or IDs may be used, as well as ranges.").Default("all").String()
excludeDetectors=cli.Flag("exclude-detectors","Comma separated list of detector types to exclude. Protobuf name or IDs may be used, as well as ranges. IDs defined here take precedence over the include list.").String()
gitScan=cli.Command("git","Find credentials in git repositories.")
gitScanURI=gitScan.Arg("uri","Git repository URL. https:// or file:// schema expected.").Required().String()
gitScanURI=gitScan.Arg("uri","Git repository URL. https://, file://, or ssh:// schema expected.").Required().String()
gitScanIncludePaths=gitScan.Flag("include-paths","Path to file with newline separated regexes for files to include in scan.").Short('i').String()
gitScanExcludePaths=gitScan.Flag("exclude-paths","Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
gitScanExcludeGlobs=gitScan.Flag("exclude-globs","Comma separated list of globs to exclude in scan. This option filters at the `git log` level, resulting in faster scans.").String()
gitScanSinceCommit=gitScan.Flag("since-commit","Commit to start scan from.").String()
gitScanBranch=gitScan.Flag("branch","Branch to scan.").String()
gitScanMaxDepth=gitScan.Flag("max-depth","Maximum depth of commits to scan.").Int()
@@ -54,32 +70,63 @@ var (
_=gitScan.Flag("entropy","No-op flag for backwards compat.").Bool()
_=gitScan.Flag("regex","No-op flag for backwards compat.").Bool()
githubScan=cli.Command("github","Find credentials in GitHub repositories.")
githubScanRepos=githubScan.Flag("repo",`GitHub repository to scan. You can repeat this flag. Example: "https://github.com/dustin-decker/secretsandstuff"`).Strings()
githubScanOrgs=githubScan.Flag("org",`GitHub organization to scan. You can repeat this flag. Example: "trufflesecurity"`).Strings()
githubScanRepos=githubScan.Flag("repo",`GitHub repository to scan. You can repeat this flag. Example: "https://github.com/dustin-decker/secretsandstuff"`).Strings()
githubScanOrgs=githubScan.Flag("org",`GitHub organization to scan. You can repeat this flag. Example: "trufflesecurity"`).Strings()
githubScanToken=githubScan.Flag("token","GitHub token. Can be provided with environment variable GITHUB_TOKEN.").Envar("GITHUB_TOKEN").String()
githubIncludeForks=githubScan.Flag("include-forks","Include forks in scan.").Bool()
githubIncludeMembers=githubScan.Flag("include-members","Include organization member repositories in scan.").Bool()
githubIncludeRepos=githubScan.Flag("include-repos",`Repositories to include in an org scan. This can also be a glob pattern. You can repeat this flag. Must use Github repo full name. Example: "trufflesecurity/trufflehog", "trufflesecurity/t*"`).Strings()
githubExcludeRepos=githubScan.Flag("exclude-repos",`Repositories to exclude in an org scan. This can also be a glob pattern. You can repeat this flag. Must use Github repo full name. Example: "trufflesecurity/driftwood", "trufflesecurity/d*"`).Strings()
githubScanIncludePaths=githubScan.Flag("include-paths","Path to file with newline separated regexes for files to include in scan.").Short('i').String()
githubScanExcludePaths=githubScan.Flag("exclude-paths","Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
gitlabScan=cli.Command("gitlab","Find credentials in GitLab repositories.")
gitlabScanRepos=gitlabScan.Flag("repo","GitLab repo url. You can repeat this flag. Leave empty to scan all repos accessible with provided credential. Example: https://gitlab.com/org/repo.git").Strings()
gitlabScanRepos=gitlabScan.Flag("repo","GitLab repo url. You can repeat this flag. Leave empty to scan all repos accessible with provided credential. Example: https://gitlab.com/org/repo.git").Strings()
gitlabScanToken=gitlabScan.Flag("token","GitLab token. Can be provided with environment variable GITLAB_TOKEN.").Envar("GITLAB_TOKEN").Required().String()
gitlabScanIncludePaths=gitlabScan.Flag("include-paths","Path to file with newline separated regexes for files to include in scan.").Short('i').String()
gitlabScanExcludePaths=gitlabScan.Flag("exclude-paths","Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
filesystemScan=cli.Command("filesystem","Find credentials in a filesystem.")
filesystemDirectories=filesystemScan.Flag("directory","Path to directory to scan. You can repeat this flag.").Required().Strings()
filesystemScan=cli.Command("filesystem","Find credentials in a filesystem.")
filesystemPaths=filesystemScan.Arg("path","Path to file or directory to scan.").Strings()
// DEPRECATED: --directory is deprecated in favor of arguments.
filesystemDirectories=filesystemScan.Flag("directory","Path to directory to scan. You can repeat this flag.").Strings()
// TODO: Add more filesystem scan options. Currently only supports scanning a list of directories.
// filesystemScanIncludePaths = filesystemScan.Flag("include-paths", "Path to file with newline separated regexes for files to include in scan.").Short('i').String()
// filesystemScanExcludePaths = filesystemScan.Flag("exclude-paths", "Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
filesystemScanIncludePaths=filesystemScan.Flag("include-paths","Path to file with newline separated regexes for files to include in scan.").Short('i').String()
filesystemScanExcludePaths=filesystemScan.Flag("exclude-paths","Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
s3Scan=cli.Command("s3","Find credentials in S3 buckets.")
s3ScanKey=s3Scan.Flag("key","S3 key used to authenticate.").String()
s3ScanSecret=s3Scan.Flag("secret","S3 secret used to authenticate.").String()
s3ScanCloudEnv=s3Scan.Flag("cloud-environment","Use IAM credentials in cloud environment.").Bool()
s3ScanBuckets=s3Scan.Flag("bucket","Name of S3 bucket to scan. You can repeat this flag.").Strings()
s3Scan=cli.Command("s3","Find credentials in S3 buckets.")
s3ScanKey=s3Scan.Flag("key","S3 key used to authenticate. Can be provided with environment variable AWS_ACCESS_KEY_ID.").Envar("AWS_ACCESS_KEY_ID").String()
s3ScanSecret=s3Scan.Flag("secret","S3 secret used to authenticate. Can be provided with environment variable AWS_SECRET_ACCESS_KEY.").Envar("AWS_SECRET_ACCESS_KEY").String()
s3ScanSessionToken=s3Scan.Flag("session-token","S3 session token used to authenticate temporary credentials. Can be provided with environment variable AWS_SESSION_TOKEN.").Envar("AWS_SESSION_TOKEN").String()
s3ScanCloudEnv=s3Scan.Flag("cloud-environment","Use IAM credentials in cloud environment.").Bool()
s3ScanBuckets=s3Scan.Flag("bucket","Name of S3 bucket to scan. You can repeat this flag.").Strings()
gcsScan=cli.Command("gcs","Find credentials in GCS buckets.")
gcsProjectID=gcsScan.Flag("project-id","GCS project ID used to authenticate. Can NOT be used with unauth scan. Can be provided with environment variable GOOGLE_CLOUD_PROJECT.").Envar("GOOGLE_CLOUD_PROJECT").String()
gcsCloudEnv=gcsScan.Flag("cloud-environment","Use Application Default Credentials, IAM credentials to authenticate.").Bool()
gcsServiceAccount=gcsScan.Flag("service-account","Path to GCS service account JSON file.").ExistingFile()
gcsWithoutAuth=gcsScan.Flag("without-auth","Scan GCS buckets without authentication. This will only work for public buckets").Bool()
gcsAPIKey=gcsScan.Flag("api-key","GCS API key used to authenticate. Can be provided with environment variable GOOGLE_API_KEY.").Envar("GOOGLE_API_KEY").String()
gcsIncludeBuckets=gcsScan.Flag("include-buckets","Buckets to scan. Comma seperated list of buckets. You can repeat this flag. Globs are supported").Short('I').Strings()
gcsExcludeBuckets=gcsScan.Flag("exclude-buckets","Buckets to exclude from scan. Comma separated list of buckets. Globs are supported").Short('X').Strings()
gcsIncludeObjects=gcsScan.Flag("include-objects","Objects to scan. Comma separated list of objects. you can repeat this flag. Globs are supported").Short('i').Strings()
gcsExcludeObjects=gcsScan.Flag("exclude-objects","Objects to exclude from scan. Comma separated list of objects. You can repeat this flag. Globs are supported").Short('x').Strings()
gcsMaxObjectSize=gcsScan.Flag("max-object-size","Maximum size of objects to scan. Objects larger than this will be skipped. (Byte units eg. 512B, 2KB, 4MB)").Default("10MB").Bytes()
syslogScan=cli.Command("syslog","Scan syslog")
syslogAddress=syslogScan.Flag("address","Address and port to listen on for syslog. Example: 127.0.0.1:514").String()
syslogProtocol=syslogScan.Flag("protocol","Protocol to listen on. udp or tcp").String()
syslogTLSCert=syslogScan.Flag("cert","Path to TLS cert.").String()
syslogTLSKey=syslogScan.Flag("key","Path to TLS key.").String()
syslogFormat=syslogScan.Flag("format","Log format. Can be rfc3164 or rfc5424").String()
circleCiScanToken=circleCiScan.Flag("token","CircleCI token. Can also be provided with environment variable").Envar("CIRCLECI_TOKEN").Required().String()
)
funcinit(){
@@ -94,22 +141,26 @@ func init() {
cli.Version("trufflehog "+version.BuildVersion)
cmd=kingpin.MustParse(cli.Parse(os.Args[1:]))
if*jsonOut{
logrus.SetFormatter(&logrus.JSONFormatter{})
}
switch{
case*trace:
logrus.SetLevel(logrus.TraceLevel)
logrus.Debugf("running version %s",version.BuildVersion)
log.SetLevel(5)
case*debug:
logrus.SetLevel(logrus.DebugLevel)
logrus.Debugf("running version %s",version.BuildVersion)
data:[]byte(fmt.Sprintf("You can find a aha secret %s within",secret)),
verify:true,
},
want:[]detectors.Result{
{
DetectorType:detectorspb.DetectorType_Aha,
Verified:true,
},
},
wantErr:false,
},
{
name:"found, unverified",
s:Scanner{},
args:args{
ctx:context.Background(),
data:[]byte(fmt.Sprintf("You can find a aha secret %s within but not valid",inactiveSecret)),// the secret would satisfy the regex but not pass validation
data:[]byte(fmt.Sprintf("You can find a alchemy secret %s within",secret)),
verify:true,
},
want:[]detectors.Result{
{
DetectorType:detectorspb.DetectorType_Alchemy,
Verified:true,
},
},
wantErr:false,
},
{
name:"found, unverified",
s:Scanner{},
args:args{
ctx:context.Background(),
data:[]byte(fmt.Sprintf("You can find a alchemy secret %s within but not valid",inactiveSecret)),// the secret would satisfy the regex but not pass validation
data:[]byte(fmt.Sprintf("Alibaba keys are here: %s\n %s within",id,secretInactive)),
data:[]byte(fmt.Sprintf("You can find a alibaba secret %s within alibaba %s but not valid",inactiveSecret,id)),// the secret would satisfy the regex but not pass validation
@@ -58,7 +56,7 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
res,err:=client.Do(req)
iferr==nil{
deferres.Body.Close()
bodyBytes,err:=ioutil.ReadAll(res.Body)
bodyBytes,err:=io.ReadAll(res.Body)
iferr!=nil{
continue
}
@@ -77,5 +75,9 @@ func (s Scanner) FromData(ctx context.Context, verify bool, data []byte) (result
results=append(results,s1)
}
returndetectors.CleanResults(results),nil
returnresults,nil
}
func(sScanner)Type()detectorspb.DetectorType{
returndetectorspb.DetectorType_Allsports
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Bill Rich
