def91bd138
Create devcontainer.json for codespaces
2023-08-01 08:12:41 -07:00
69021f59c5
Refactor git source to allow ScanOptions and use source in engine ( #1518 )
...
* Refactor git source to allow ScanOptions and use source in engine
Refactor the Chunks method of the git Source to call out to two helper
methods: scanRepos and scanDirs which scans s.conn.Repositories and
s.conn.Directories respectively. The only notable change in behavior is
that a credential is no longer necessary if there are no
s.conn.Repositories to scan.
* Preserve ScanGit functionality of not cleaning up temporary files
2023-08-01 09:52:02 -05:00
b8c43ea58f
Fix VirusTotal deetector ( #1585 )
2023-08-01 05:41:18 -07:00
a4b1fb7752
create hidden debug flag to disable overseer ( #1582 )
...
* add in new debug flag
* keep localdev local
2023-07-31 22:03:59 -04:00
7d2f126411
add tri-state verification to mongodb detector ( #1575 )
2023-07-31 18:23:35 -04:00
5043fc8756
[bug] - Fix unlocking an unlocked mutex ( #1583 )
...
* use correct mutext.
* remove unused fxn.
2023-07-31 14:06:41 -07:00
661c6b47b7
[bug] - fix shodan detector ( #1579 )
...
* fix shodan detector.
* fix import order.
2023-07-31 11:12:52 -07:00
eb00d0d4e1
[bug] - fix data races ( #1577 )
...
* fix data race.
* Add test and fix additional data race.
* address comments.
2023-07-31 11:12:38 -07:00
406ce7bc55
Replace magic strings with const ( #1568 )
...
* Add normalize repo for azure.
* remove zero values from test cases.
* use const.
* remove azure logic.
2023-07-31 11:12:26 -07:00
5e7a6ca11c
Concurrent detection ( #1580 )
...
* Run detection on each chunk concurrently.
* Add printer functionality.
* Add logic for dedupe.
* cleanup.
* Moddify number of notifier workers.
* Add comment.
* move consts into fxn.
* buffer resutls chan.
* fix test.
* address comments.
* return an error from Finish.
* fix test.
* fix test.
* linter.
* check err.
* address comments.
2023-07-31 11:12:08 -07:00
b54683acb9
gitparse: Use an object for currentDiff ( #1573 )
...
* gitparse: Use an object for currentDiff instead of a pointer
* gitparse: Use an object for currentCommit instead of a pointer
* Revert "gitparse: Use an object for currentCommit instead of a pointer"
This reverts commit c5f0708b4a
2023-07-31 11:39:14 -05:00
32e3f1f015
Fix pubnub regular expression ( #1565 )
...
One of the sub-groups of the UUIDv4 was missing the characters 0-9.
2023-07-31 11:37:25 -05:00
a07b6664f8
Support fatal errors in job reports ( #1562 )
...
* Support fatal errors in job reports
* WIP: JobReporter and JobInspector
* WIP: JobReportHook and JobReportRef
* Add ChunkError type and asyncRun helper method
* Rename JobReport to JobProgress
* Return a closed channel from Done when the JobProgress is nil
* Comment catchFirstFatal function
2023-07-31 11:28:30 -05:00
61bee6c8b1
Identify transient AWS verification failures ( #1563 )
...
It turns out that GetCallerIdentity returns a surprising quantity of transient, false-negative 403 responses that carry the SignatureDoesNotMatch error reason. I don't know why this is happening, but their transient nature makes them indeterminate verification failures and they should be flagged as such. The AWS detector has therefore been modified to specifically look for the InvalidClientTokenId error reason in 403 responses and mark all other responses as indeterminate.
In addition to the functional changes this PR contains some updates to the test code that allow us to test them.
2023-07-31 12:06:11 -04:00
ad57de50cd
Do not nest transports for Github installation client ( #1564 )
...
#1454 modified one of the Github enumeration code paths in a way that broke an integration test by causing one client's transport to be used for the construction of a different client, causing authentication failures. This saves the original transport for use, fixing the test.
2023-07-31 11:31:16 -04:00
e0faac8d1c
Fix runtime error when scanning Gist comments ( #1552 )
...
* fix(github): fix runtime error from gist comments
* fix(github): add flag to scan Gist comments
2023-07-31 08:57:42 -05:00
070014f380
Initialize the default logger to output to stderr ( #1569 )
2023-07-31 03:26:19 -05:00
10b6e2898d
Increase log level of engine messages ( #1576 )
2023-07-28 14:30:43 -07:00
6bd48583ae
Fix gitparse from panicking on a nil-pointer ( #1570 )
2023-07-28 11:15:02 -05:00
431d26f5fa
move false positive check in alchemy detector ( #1532 )
...
This PR makes the Alchemy detector run its known false positive check even if verification is disabled. This isn't the most important detector but it's the template for new ones so getting a good pattern nailed down is important.
Moving the check allowed me to rewrite the determinacy logic to hopefully be more clear.
2023-07-28 11:36:02 -04:00
9f3809f19e
gdrive proto change ( #1566 )
2023-07-28 10:38:51 -04:00
e391e89f3e
Initial implementation of JobReport with SourceManager usage ( #1557 )
...
* Initial implementation of JobReport with SourceManager usage
* Limit concurrent units
* Only save the last JobReport per handle
2023-07-27 10:49:56 -05:00
3897454dbb
add merge support ( #1561 )
2023-07-27 09:24:49 -05:00
ade5d91d5c
Add azure repos protos. ( #1559 )
2023-07-26 19:53:10 -07:00
84d4bf0d51
Override broken dependency version ( #1558 )
2023-07-26 16:13:02 -07:00
f925da7cea
fix(mongodb): detect CosmoDB access keys ( #1511 )
...
https://learn.microsoft.com/en-us/microsoft-365/compliance/sit-defn-azure-cosmos-db-account-access-key?view=o365-worldwide
2023-07-26 16:50:12 -05:00
e0011ac08e
Bump github.com/go-git/go-git/v5 from 5.8.0 to 5.8.1 ( #1554 )
...
Bumps [github.com/go-git/go-git/v5](https://github.com/go-git/go-git ) from 5.8.0 to 5.8.1.
- [Release notes](https://github.com/go-git/go-git/releases )
- [Commits](https://github.com/go-git/go-git/compare/v5.8.0...v5.8.1 )
---
updated-dependencies:
- dependency-name: github.com/go-git/go-git/v5
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-26 14:13:12 -07:00
7a29f8a514
Bump github.com/bradleyfalzon/ghinstallation/v2 from 2.4.0 to 2.6.0 ( #1503 )
...
Bumps [github.com/bradleyfalzon/ghinstallation/v2](https://github.com/bradleyfalzon/ghinstallation ) from 2.4.0 to 2.6.0.
- [Release notes](https://github.com/bradleyfalzon/ghinstallation/releases )
- [Commits](https://github.com/bradleyfalzon/ghinstallation/compare/v2.4.0...v2.6.0 )
---
updated-dependencies:
- dependency-name: github.com/bradleyfalzon/ghinstallation/v2
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-26 09:03:55 -07:00
46823f77c9
feat(github): clarify comment log statement ( #1553 )
2023-07-26 09:40:30 -05:00
10f0963bc9
Add SourceManager tests for Run and Wait methods ( #1530 )
...
* Miscellaneous SourceManager updates
* Own the chunks channel instead of accepting it as an input
* Add Chunks and Wait methods
* Fix bug in Enroll so it actually returns the handle
* Add context.Context parameter to the SourceInitFunc type
* Add SourceManager tests for Run and Wait methods
* Rename man variables to mgr
2023-07-26 00:48:28 -05:00
35b332c498
Bump github.com/google/go-containerregistry from 0.14.0 to 0.15.2 ( #1504 )
...
Bumps [github.com/google/go-containerregistry](https://github.com/google/go-containerregistry ) from 0.14.0 to 0.15.2.
- [Release notes](https://github.com/google/go-containerregistry/releases )
- [Changelog](https://github.com/google/go-containerregistry/blob/main/.goreleaser.yml )
- [Commits](https://github.com/google/go-containerregistry/compare/v0.14.0...v0.15.2 )
---
updated-dependencies:
- dependency-name: github.com/google/go-containerregistry
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Bill Rich <bill.rich@gmail.com >
2023-07-25 20:33:18 -07:00
c814ae4d83
Bump github.com/xanzy/go-gitlab from 0.86.0 to 0.88.0 ( #1522 )
...
Bumps [github.com/xanzy/go-gitlab](https://github.com/xanzy/go-gitlab ) from 0.86.0 to 0.88.0.
- [Changelog](https://github.com/xanzy/go-gitlab/blob/master/releases_test.go )
- [Commits](https://github.com/xanzy/go-gitlab/compare/v0.86.0...v0.88.0 )
---
updated-dependencies:
- dependency-name: github.com/xanzy/go-gitlab
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Bill Rich <bill.rich@gmail.com >
2023-07-25 20:31:29 -07:00
2290954b02
fix(github): use apiEndpoint for basic or no auth ( #1454 )
2023-07-25 20:03:08 -07:00
f48a635c34
feat: update gitparse logic ( #1486 )
2023-07-25 17:52:34 -05:00
91cbca941a
Fix data race in context wrapper library ( #1546 )
2023-07-25 17:09:36 -05:00
1a1977f7e6
case insensitive ( #1547 )
2023-07-25 17:01:15 -05:00
f39303495a
Add commitsScanned metrics ( #1533 )
...
* Add commitsScanned metrics
* Just keep commit count
2023-07-25 11:31:01 -07:00
b5b01d3eba
[chore] - optimize chunker ( #1535 )
...
* Use chunkbytes that includes the size of peek.
* linter.
* continue.
* add TotalChunkSize const.
2023-07-24 19:30:29 -07:00
85f363f093
init ( #1538 )
2023-07-24 19:09:57 -05:00
302c3ce8d1
Downgrade couchbase dep ( #1543 )
...
* update couchbase dep
* downgrade couchbase dep
2023-07-24 17:00:13 -07:00
fd6c281589
update couchbase dep ( #1540 )
2023-07-24 16:27:11 -07:00
93c561f324
Add match boundary to okta regular expressions ( #1531 )
2023-07-24 10:52:50 -05:00
fab80445d1
continue scanning on detector / decoder panic ( #863 )
2023-07-24 07:34:43 -07:00
9e0a2e9ddd
[chore] - Remove password info from log ( #1528 )
...
* Remove password info from log.
* update.
* one more.
2023-07-22 20:25:45 -07:00
69515bb7ac
Correctly route pprof endpoint ( #1527 )
2023-07-21 22:45:27 -05:00
91c5472876
Implement SourceManager basics ( #1515 )
...
* Implement SourceManager basics
* Rename identifiers and add a default headlessAPI implementation
* Rewrite to use SourceInitFunc
* Update variable name to accurately reflect its value
2023-07-21 15:20:25 -05:00
a613bbb979
[chore] Remove parent manipulation in context package ( #1525 )
...
The ability to set the parent allowed creating context cycles which
shouldn't be allowed, or at the very least have unintuitive behavior.
2023-07-21 13:51:51 -05:00
ebf1038392
Support indeterminacy in alchemy and update detector docs ( #1510 )
2023-07-21 14:50:14 -04:00
be68eb044a
Decrease frequency of dependabot alerts to monthly ( #1524 )
...
* Barely useful
* Update dependabot.yml
2023-07-21 11:22:03 -05:00
06a562688d
capture json error ( #1509 )
2023-07-21 10:44:47 -05:00
Dustin Decker