A previous commit (5d0196957f) added .jar/.war/.ear files to the ignored extensions list, but these are archive files that we can scan, so we shouldn't exclude them.
This PR expands the list of excluded file extensions to contain images and other binary files. These files can technically contain secrets, but need decoding to properly be handled, and we don't have any such decoding yet. Down the road if we want to add it we can.
* Add generic glob filter
* Make nil filters safe
* Include glob in error
* Use better example for exclude and include test
* Allow user to configure the ambiguous case
* Rename Pass to ShouldInclude and invert logic
* Test default *Filter and Filter have the same behavior of allow
* Add property based tests
* Remove configuration for the not found ambiguous case
It turns out that GetCallerIdentity returns a surprising quantity of transient, false-negative 403 responses that carry the SignatureDoesNotMatch error reason. I don't know why this is happening, but their transient nature makes them indeterminate verification failures and they should be flagged as such. The AWS detector has therefore been modified to specifically look for the InvalidClientTokenId error reason in 403 responses and mark all other responses as indeterminate.
In addition to the functional changes this PR contains some updates to the test code that allow us to test them.
* Add CancellableWrite helper function
* Create SourceUnitEnumerator interface and EnumerationResult struct
* Implement SourceUnitEnumerator for the filesystem Source
* Omit explicit zero values
* verify response body with expected keywords
* remove debug log
* add extra test case
* migrate from ioutil to io
* close body and only check for one keyword
* cleanup
* init
* add detector type
* rotate leaked credentials
* tighten up username pattern
* isolated prefixregex as overrriding new line stuff
* passwordPat working now
* add username test
* fix edge case
* cleanup
* make linter happy
* make linter happy rd 2
* skip error logging
* fix test
* add password regex helper func
* make test more robust
* cleanup PR
* remove comments
* clarify prepend rationale
* Adding missing flags to Readme
* Use retryableHttpClient by default for GitHub
* Adding repoUrl for scanning time log
* Use WithField instead of WithFields
* Updating README with lasted --help output
* Use a config struct when scanning and engine source.
* fix tests.
* Move test_helpers to the sources pkg.
* Handle ScanGit error in tests.
* adderss comments.
* Use functional options.
* Remove temp var.
* Add better var names for the setup functions for each config.
* Remove unused var.
* fix error logs.
* fix error logs.
* single line.
* remove blank lines.
* added common regex patterns for detectors
* For HexPattern
* enhancements
* used parseInt
* enhancement
* enhanced regex for email and subdomain
* enhancement for email pattern
* update pattern and detector
Co-authored-by: Roxanne Tampus <roxannetampus02@gmail.com>
* Small amount of code clean up.
* Rename sem to concurrency for better readability and to remove an extra comment.
* fix stashing issue.
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
* Add missing pagination on github calls
Includes some refactoring to improve readability and code reuse.
* Close response body and handle rate limit
* Re-include support for including users as repos to github scans
* Fix gist test to match new func signature
* Add current test name to logging
* Support username as org use case
* Also include no-auth user as org
Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
* include and exclude paths gitscan args
Add support for include_paths and exclude_paths arguments when scanning
git sources.
* Improve variable name
Co-authored-by: Bill Rich <hrich@Bills-MacBook-Pro.local>
ahrav