The source manager initialization function was defined as `sourceID`
followed by `jobID`, while the source initialization function is the
reverse. This is confusing and easy to mix up since the parameters are
the same type.
This commit adds a test to make sure the source manager initializes in
the correct order, but it doesn't prevent the library user to make the
same mistake. We may want to consider using different types.
* add exportable validate function for github
* update validator
* use the context
* gate to prevent panic
* wrap error with context
* wrap error with context for basic auth and unauth
* add role assumption for s3 source
* refactor role assumption to repeatable string
user can pass array of roles to assume
* refactor s3 chunks to handle passed roleARNs
* add role-session name
use timestamp to make dynamic
* add docstring for rolearn strings()
* make sure role ars are passed into source
* refactor role assumption functionality
break s3 bucket scanning into sep. function
* add log check on assume role
* fix role iteration
- Make sure s3 struct is populated with roles
- add separate new client instantiation for role-based access
- iterates through each role
* add comment
* protobuf revert for merge
* re-run make proto
* lint cleanup
* cleanup TODOs
* drop redundant switch case in assumerole client
* use less verbose 'ctx' designator
* breakout functionality from Chunks
- separate functions for:
- enumerating buckets to scan
- scanning objects within the buckets
* remake protobuf defs
* allow scan to continue on single bucket err
* add readme docs
* minor fixups
With the introduction of the SourceManager, the chunks channel became
private and read-only. This provides a method to write chunks into the
channel as we transition away from needing to do that.
* Use custom context for archive handler of specialized archives.
* fix arg.
* fix test.
* use re-reader.
* use re-reader.
* Update error and comments.
* Add better error handling.
* update.
* setup
* update time out case to return detector result
* fix
* remove unneeded comment
* remove debug print
* cleanup
* more robust error handling
* reflect new detector template changes
* fixes
* mark response body check err as indeterminate
* Add handler for .deb file formats.
* Add handler for .rpm file formats.
* update.
* move logic to general archive handler.
* update const.
* Add compile time guard.
* Remove redundant parens.
* Add checks to make sure we have the tools installed to extract arhives.
* Limit size of temp file for arhive reading.
* handle nested archives.
* add comment.
* use consistent name for tempEnv -> env
* fix handler fxn signature.
* added functionality to scan docker images with digests instead of tags
* cleaned import statement
* added unit test for baseAndTag parsing + remote digest scan
Verification of an Alchemy secret requires putting the candidate secret directly into a URL. This makes the URL potentially sensitive, and if the request fails, we don't want to save it anywhere that might inadvertently get logged elsewhere - like the resulting error message. (Despite verification failing, this error message is only saved if the failure is indeterminate, which means that the secret might actually be live.)
* Init attempt at tui with bubble tea.
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Add starting and source selection options
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Rewrite models into a state machine
* Update source descriptions
* Make subpages implement tea.Model
* Rename page0 and page1 to be more descriptive
* Adjust styling and adding color consts
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Add helper generic function to call Update and type cast
* Setup plumbing for source configuration page
* Use CLI introspection for source configuration (WIP)
* Experiment with table view
* Replace table with form fields
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Change 🔒 to 💸
* Copy components from soft-serve
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Copy styles from soft-serve
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Copy common from soft-serve
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Refactor into pages
This is still a WIP, but the main structure is there.
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Trying out selector for wizard intro
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Use selector with custom View
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Change Item to be an enum
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add link pages
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Update source select to use selector
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Delete source configure page and add blank tabs
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add tab placeholder pages for configurationi
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Added headers and style to each tab
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Update with new sources
* Remove kingpin attribute from SourceItem
* Add basic form field and source structuring
* Hookup git form fields with an underlying textinput component
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Update forms for git and github
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Add labels per text input
* Add sources and adjust styling
* add basic trufflehog configuration page
* Add skip button to textinputs component
* Emit and handle textinputs skip/submit button commands
* Don't quit when q is pressed on the sourceConfigurePage
* Build trufflehog command based on source config vals
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Build flags based on truffle config inputs
* Update summary section
* Add generated truffle fields
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* update summary to correctly print info
* Go back a page when escape key is pressed
* WIP run page list
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Allow running trufflehog from the run page
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add option to view help docs
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* comment out unused styles and remove unused types
* Capitalize H in TruffleHog
* remove unneeded fmt.Sprintf
---------
Co-authored-by: mcastorina <m.castorina93@gmail.com>
This PR implements tri-state verification in the FTP detector. The verification timeout was made injectable to support a new test case. Some test cases that had already been broken have been fixed as well.
* Add common chunker.
* add comment.
* use better config name.
* Add common chunk reader to s3.
* Add common chunk reader to git, gcs, circleci.
* revert gcs.
* revert gcs.
* fix chunker.
* revert gcs.
* update cancellablewrite.
* revert impl.
* update to remove totalsize.
* Fix my goof.
* Use unified struct in chunkreader.
* return err instead of logging and returning.
* rename error to err.
* only send single ChunkResult even if there is an error and chunkBytes.
* fix logic.
* Add SourceManager to Engine struct
* Update Engine methods to use the SourceManager
* Fix GCS test
The original was testing that `Init()` errors weren't surfaced in
`Finish()`, but the `SourceManager` changed that behavior.
* JobProgress race fixes
* Add contextual values
* Remove unused code
* Add debug logs
* Rename WithConcurrency to WithConcurrentSources
* Always forward chunks to the output chunks channel
This PR implements tri-state verification for the LDAP detector. This implementation looks for network errors to explicitly flag as indeterminate, rather than authentication errors to explicitly flag as determinate; this is because the error that occurs from authentication failures doesn't appear to have its own type and I didn't want to have to match on the error message text.