Commit Graph

1905 Commits

Author SHA1 Message Date
joeleonjr f7cf80ee85 Merge branch 'main' into docker-tar-identification-from-filesystem-2 2023-08-23 15:49:10 -04:00
Joe Leon e523b69a4c reverted size 2023-08-23 15:47:27 -04:00
Joe Leon f4119799d9 cleaned up code 2023-08-23 15:46:02 -04:00
Joe Leon 6b8ce2fba2 saving test case progress 2023-08-23 15:43:37 -04:00
Joe Leon 95f65281ea moved docker base/img hint to sources.go 2023-08-23 11:46:04 -04:00
Joe Leon 5f4f513014 refactored to call docker chunk scanner directly 2023-08-23 10:49:58 -04:00
ahrav 9ae72308be Include the job ID in a chunk (#1652)
* Include the job ID in a source's chunk.

* address comments.

* address comments.
2023-08-22 14:00:27 -07:00
Zubair Khan fd00d2b30b add rate limit and consumption metrics for GitHub (#1651)
* add rate limit and consumption metrics

* incrment after each repo scanned

* update repo scanned label name
2023-08-22 15:01:59 -04:00
Cody Rose 059ea23a72 update s3 test bucket (#1649)
We're switching our S3 source test account over to a different one, which means we have to change the bucket name.
2023-08-22 12:43:38 -04:00
Miccah 5cfbde783f Fix reversed ordering of arguments (#1648)
The source manager initialization function was defined as `sourceID`
followed by `jobID`, while the source initialization function is the
reverse. This is confusing and easy to mix up since the parameters are
the same type.

This commit adds a test to make sure the source manager initializes in
the correct order, but it doesn't prevent the library user to make the
same mistake. We may want to consider using different types.
2023-08-22 07:55:56 -07:00
Zubair Khan 9a13c74a35 add thog CLI support for GitHub config validate (#1626)
* add exportable validate function for github

* update validator

* use the context

* gate to prevent panic

* wrap error with context

* wrap error with context for basic auth and unauth
2023-08-22 10:22:39 -04:00
Cody Rose dbb2c2e319 wait before finishing s3 test (#1647)
The S3 source test verifies that chunking has completed, but it didn't actually wait for completion first, leading to non-deterministic test failures.
2023-08-21 12:36:36 -04:00
Joe Leon 2d46cf1f74 temp increase file size in maxreader, handler improvements 2023-08-21 12:15:34 -04:00
Cody Rose ed06217862 Add tri-state verification to sqlserver detector (#1624)
This is a different detector than the general JDBC detector.
2023-08-21 10:05:45 -04:00
ahrav d51e3b6d83 Only scan gist comments or repo comments. (#1646) 2023-08-20 11:38:28 -07:00
Joe Leon aa461ae806 attempting to fix archive test 2023-08-18 21:12:54 -04:00
joeleonjr a55237376a Merge branch 'main' into docker-tar-identification-from-filesystem-2 2023-08-18 21:10:54 -04:00
Joe Leon 5b9a862ffa adjusted contexts in archive tests 2023-08-18 21:01:37 -04:00
Joe Leon e01f6d0257 switched back log context naming 2023-08-18 16:48:07 -04:00
Joe Leon af83970f77 reworked tarred docker img scanning 2023-08-18 13:07:14 -04:00
Mike Vanbuskirk 64dd49f9ce add role assumption for s3 source (#1477)
* add role assumption for s3 source

* refactor role assumption to repeatable string

user can pass array of roles to assume

* refactor s3 chunks to handle passed roleARNs

* add role-session name

use timestamp to make dynamic

* add docstring for rolearn strings()

* make sure role ars are passed into source

* refactor role assumption functionality

break s3 bucket scanning into sep. function

* add log check on assume role

* fix role iteration

- Make sure s3 struct is populated with roles
- add separate new client instantiation for role-based access
- iterates through each role

* add comment

* protobuf revert for merge

* re-run make proto

* lint cleanup

* cleanup TODOs

* drop redundant switch case in assumerole client

* use less verbose 'ctx' designator

* breakout functionality from Chunks

- separate functions for:
- enumerating buckets to scan
- scanning objects within the buckets

* remake protobuf defs

* allow scan to continue on single bucket err

* add readme docs

* minor fixups
2023-08-17 20:30:20 -04:00
ahrav 0ae8cf5d35 [bug] - handle IOOR panic (#1639)
* handle IOOR panic.

* use a better fxn name.

* increae timeout for test to compete.

* simplify code and add test.

* do it for miccah.
2023-08-17 15:47:11 -07:00
ahrav 55b9d48e0d updat test file. (#1637) 2023-08-17 10:16:25 -07:00
ahrav f3c2d5e6c7 [bug] - Correctly reset reader before handling archive chunk data (#1636)
* Correctly reset reader before handling archive chunk.

* stop the re-reader.
2023-08-17 10:04:43 -07:00
ahrav b8bb94f2b1 [bug] - copy chunk before sending on chunksChan (#1633)
* Redclare chunk before sending on chunksChan.

* add integration test.

* update test.
2023-08-16 16:36:38 -07:00
Miccah fae54c7ffa Add ScanChunk to allow injecting Chunks into the SourceManager's channel (#1634)
With the introduction of the SourceManager, the chunks channel became
private and read-only. This provides a method to write chunks into the
channel as we transition away from needing to do that.
2023-08-16 16:09:23 -07:00
Zubair Khan db89e345d7 correct logging output for github comments and add oss flags (#1632)
* correct logging output

* add flags

* respect oss cli flags for github comment scanning

* improve copy
2023-08-16 18:23:59 -04:00
ahrav e0db575d4a [chore] - Use custom context for archive handler of specialized archives (#1629)
* Use custom context for archive handler of specialized archives.

* fix arg.

* fix test.

* use re-reader.

* use re-reader.

* Update error and comments.

* Add better error handling.

* update.
2023-08-16 13:52:55 -07:00
Zubair Khan 62d359eba4 add salesforce detector (#1608)
* setup

* update time out case to return detector result

* fix

* remove unneeded comment

* remove debug print

* cleanup

* more robust error handling

* reflect new detector template changes

* fixes

* mark response body check err as indeterminate
2023-08-16 10:42:04 -04:00
ahrav 6ad5659334 Integration of SpecializedHandler for Enhanced Archive Processing (#1625)
* Add handler for .deb file formats.

* Add handler for .rpm file formats.

* update.

* move logic to general archive handler.

* update const.

* Add compile time guard.

* Remove redundant parens.

* Add checks to make sure we have the tools installed to extract arhives.

* Limit size of temp file for arhive reading.

* handle nested archives.

* add comment.

* use consistent name for tempEnv -> env

* fix handler fxn signature.
2023-08-15 16:08:55 -07:00
Hon ea8c21051d fix github org placeholder (#1627) 2023-08-15 15:27:52 -07:00
Brandon Yan 2de5c0b217 change verification endpoint (#1611) 2023-08-14 11:50:11 -07:00
Zubair Khan ea6e8b6bb5 add huggingface detector (#1621)
* init huggingface detector

* completed test
2023-08-14 14:22:04 -04:00
Miccah eae66ccf7e Refactor FragmentLineOffset to match multiline secrets (#1612)
* Refactor FragmentLineOffset to match multiline secrets

* Add tests and benchmarks

* Use bytes.Count and fix an ignore tag edge case
2023-08-14 10:51:41 -07:00
Cody Rose 09795c3591 fix alchemy test error message (#1622) 2023-08-14 13:12:46 -04:00
joeleonjr fa9469cfc7 Docker scanning by digest (#1615)
* added functionality to scan docker images with digests instead of tags

* cleaned import statement

* added unit test for baseAndTag parsing + remote digest scan
2023-08-11 16:53:12 -05:00
ahrav e894540632 Use the common chunker for scanning the filesystem source (#1619)
* Use the common chunker for scanning the filesystem source.

* remove unused conts.

* add test.
2023-08-11 13:40:10 -07:00
Cody Rose c9f58b3780 Support indeterminate verification in Gitlab detector (#1613) 2023-08-11 13:36:01 -04:00
Cody Rose 62cbef5292 stop saving alchemy url (#1614)
Verification of an Alchemy secret requires putting the candidate secret directly into a URL. This makes the URL potentially sensitive, and if the request fails, we don't want to save it anywhere that might inadvertently get logged elsewhere - like the resulting error message. (Despite verification failing, this error message is only saved if the failure is indeterminate, which means that the secret might actually be live.)
2023-08-11 10:09:05 -04:00
Cody Rose 18f854d142 Add tri-state verification to pubnub publish key detector (#1616) 2023-08-11 10:08:26 -04:00
Cody Rose 2570fdc244 fix error msg in alchemy test (#1617) 2023-08-11 09:54:37 -04:00
Hon 47c2b6bed9 Add terminal UI (#1593)
* Init attempt at tui with bubble tea.

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Add starting and source selection options

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Rewrite models into a state machine

* Update source descriptions

* Make subpages implement tea.Model

* Rename page0 and page1 to be more descriptive

* Adjust styling and adding color consts

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Add helper generic function to call Update and type cast

* Setup plumbing for source configuration page

* Use CLI introspection for source configuration (WIP)

* Experiment with table view

* Replace table with form fields

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Change 🔒 to 💸

* Copy components from soft-serve

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Copy styles from soft-serve

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Copy common from soft-serve

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Refactor into pages

This is still a WIP, but the main structure is there.

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Trying out selector for wizard intro

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Use selector with custom View

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Change Item to be an enum

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Add link pages

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Update source select to use selector

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Delete source configure page and add blank tabs

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Add tab placeholder pages for configurationi

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Added headers and style to each tab

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Update with new sources

* Remove kingpin attribute from SourceItem

* Add basic form field and source structuring

* Hookup git form fields with an underlying textinput component

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Update forms for git and github

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Add labels per text input

* Add sources and adjust styling

* add basic trufflehog configuration page

* Add skip button to textinputs component

* Emit and handle textinputs skip/submit button commands

* Don't quit when q is pressed on the sourceConfigurePage

* Build trufflehog command based on source config vals

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Build flags based on truffle config inputs

* Update summary section

* Add generated truffle fields

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* update summary to correctly print info

* Go back a page when escape key is pressed

* WIP run page list

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Allow running trufflehog from the run page

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Add option to view help docs

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* comment out unused styles and remove unused types

* Capitalize H in TruffleHog

* remove unneeded fmt.Sprintf

---------

Co-authored-by: mcastorina <m.castorina93@gmail.com>
2023-08-09 13:13:55 -07:00
Cody Rose e5aeb219de implement tri-state verification in FTP detector (#1604)
This PR implements tri-state verification in the FTP detector. The verification timeout was made injectable to support a new test case. Some test cases that had already been broken have been fixed as well.
2023-08-09 09:52:23 -04:00
Bill Rich 2d2595a2e3 Move commits_scanned to ScanRepo (#1610) 2023-08-07 14:28:57 -07:00
ahrav 13999227b9 Use common chunk reader (#1596)
* Add common chunker.

* add comment.

* use better config name.

* Add common chunk reader to s3.

* Add common chunk reader to git, gcs, circleci.

* revert gcs.

* revert gcs.

* fix chunker.

* revert gcs.

* update cancellablewrite.

* revert impl.

* update to remove totalsize.

* Fix my goof.

* Use unified struct in chunkreader.

* return err instead of logging and returning.

* rename error to err.

* only send single ChunkResult even if there is an error and chunkBytes.

* fix logic.
2023-08-07 12:55:28 -07:00
Cody Rose 18b3d3dd01 Tweak template detector test code (#1609)
The test code for the Alchemy detector, which is used as a template for new detectors, had a few little warts that I cleaned up.
2023-08-07 11:04:59 -04:00
Bill Rich c2b49b060b Detect API keys without app keys (#1605) 2023-08-03 15:11:54 -07:00
Bill Rich cda88ebdf0 Adjust regex and add tests (#1602) 2023-08-03 15:11:45 -07:00
Miccah 1cd600f70f Use SourceManager in engine (#1586)
* Add SourceManager to Engine struct

* Update Engine methods to use the SourceManager

* Fix GCS test

The original was testing that `Init()` errors weren't surfaced in
`Finish()`, but the `SourceManager` changed that behavior.

* JobProgress race fixes

* Add contextual values

* Remove unused code

* Add debug logs

* Rename WithConcurrency to WithConcurrentSources

* Always forward chunks to the output chunks channel
2023-08-03 13:36:30 -05:00
Cody Rose d763097fdf implement indeterminate LDAP verification (#1574)
This PR implements tri-state verification for the LDAP detector. This implementation looks for network errors to explicitly flag as indeterminate, rather than authentication errors to explicitly flag as determinate; this is because the error that occurs from authentication failures doesn't appear to have its own type and I didn't want to have to match on the error message text.
2023-08-03 14:02:31 -04:00