7f534d0bb7
Fix off by one ( #1891 )
2023-10-17 07:02:27 -07:00
52ed87edb7
Add an option to filter unverified results using shannon entropy ( #1875 )
...
* Add an option to filter unverified results using shannon entropy
* lint
* add test, update test, and optimize
2023-10-08 19:52:28 -07:00
6affc903e1
add line to link for azure repos. ( #1801 )
2023-09-21 16:07:11 -07:00
a8c89c59b9
[bug] - fix link line ( #1793 )
...
* fix link line.
* rename.
2023-09-20 14:46:00 -07:00
47d5ddebf2
Ability to update line number in link ( #1788 )
...
* Add functionality to update a source's link in the metadata with the updated line number.
* update comment.
* add logic to engine.
* only update link for non empty links.
* add tests for bb.
2023-09-19 15:39:13 -07:00
22876f8381
replace interface{} with any. ( #1771 )
2023-09-15 04:35:15 -07:00
fdeccf06a0
cache dupes w/ different decoders ( #1754 )
...
* only cache dupes that have different decoders.
* add test.
* remove file.
* update comment.
2023-09-11 08:18:48 -07:00
fae54c7ffa
Add ScanChunk to allow injecting Chunks into the SourceManager's channel ( #1634 )
...
With the introduction of the SourceManager, the chunks channel became
private and read-only. This provides a method to write chunks into the
channel as we transition away from needing to do that.
2023-08-16 16:09:23 -07:00
eae66ccf7e
Refactor FragmentLineOffset to match multiline secrets ( #1612 )
...
* Refactor FragmentLineOffset to match multiline secrets
* Add tests and benchmarks
* Use bytes.Count and fix an ignore tag edge case
2023-08-14 10:51:41 -07:00
1cd600f70f
Use SourceManager in engine ( #1586 )
...
* Add SourceManager to Engine struct
* Update Engine methods to use the SourceManager
* Fix GCS test
The original was testing that `Init()` errors weren't surfaced in
`Finish()`, but the `SourceManager` changed that behavior.
* JobProgress race fixes
* Add contextual values
* Remove unused code
* Add debug logs
* Rename WithConcurrency to WithConcurrentSources
* Always forward chunks to the output chunks channel
2023-08-03 13:36:30 -05:00
06d2eab204
include scan duration in output log ( #1598 )
...
* add scan duration to output log.
* fix linter.
2023-08-02 11:48:29 -07:00
5043fc8756
[bug] - Fix unlocking an unlocked mutex ( #1583 )
...
* use correct mutext.
* remove unused fxn.
2023-07-31 14:06:41 -07:00
5e7a6ca11c
Concurrent detection ( #1580 )
...
* Run detection on each chunk concurrently.
* Add printer functionality.
* Add logic for dedupe.
* cleanup.
* Moddify number of notifier workers.
* Add comment.
* move consts into fxn.
* buffer resutls chan.
* fix test.
* address comments.
* return an error from Finish.
* fix test.
* fix test.
* linter.
* check err.
* address comments.
2023-07-31 11:12:08 -07:00
10b6e2898d
Increase log level of engine messages ( #1576 )
2023-07-28 14:30:43 -07:00
1a1977f7e6
case insensitive ( #1547 )
2023-07-25 17:01:15 -05:00
85f363f093
init ( #1538 )
2023-07-24 19:09:57 -05:00
fab80445d1
continue scanning on detector / decoder panic ( #863 )
2023-07-24 07:34:43 -07:00
a613bbb979
[chore] Remove parent manipulation in context package ( #1525 )
...
The ability to set the parent allowed creating context cycles which
shouldn't be allowed, or at the very least have unintuitive behavior.
2023-07-21 13:51:51 -05:00
e8b5e3cea3
Revert "[chore] Remove parent setting / getting in Context wrapper ( #1516 )" ( #1519 )
...
This reverts commit 8ec5e4916c
2023-07-20 23:31:28 -05:00
8ec5e4916c
[chore] Remove parent setting / getting in Context wrapper ( #1516 )
...
* [chore] Remove parent setting / getting in Context wrapper
* Keep the cancellable context from errgroup
2023-07-20 13:33:09 -05:00
a9213a1103
[chore] - Update loop to switch. ( #1487 )
...
* Update loop to switch.
* remove unused fxn.
2023-07-12 15:47:43 -07:00
b48ac24c46
Dedupe results ( #1479 )
...
* init 4 dedupin
* use raw rather than rawv2
* rm comment
* comments
* nits
* clean up and use rawv2 too
* add decoder order test
2023-07-11 15:48:00 -05:00
0bdd513d88
additional similarity check for base64 and plain ( #1462 )
...
* additional similarity check for base64 and plain
* use bytes equal
* move logic into util function
2023-07-10 10:12:59 -05:00
18a70b64bb
Introduce trufflehog:ignore tag feature ( #1433 )
...
* init ignore
* cleanup and add test
* update readme
2023-06-29 08:45:56 -05:00
da5301ea1e
Exit with non-zero exit code on chunk source error ( #1286 )
...
* Exit with non-zero exit code on chunk source error
* Exit with a non-zero exit code whenever we hit an error getting
chunks. Previously the error would be logged but trufflehog would exit
with a 0 (success) status code.
* fix gcs test
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com >
Co-authored-by: ahrav <ahravdutta02@gmail.com >
2023-06-26 11:39:57 -05:00
74ffbd2878
add a custom detector check for logging duplicate detector ( #1394 )
...
* add a custom detector check for logging duplicate detector
* use pb type
2023-06-13 14:49:21 -05:00
584db86031
Support line numbers in filesystem source ( #1297 )
2023-05-09 08:02:34 -07:00
cec1543894
Add utf16 decoder proto. ( #1276 )
2023-04-20 15:25:36 -07:00
dfc5a9f5db
[chore] Log possible duplicate detectors ( #1266 )
...
* [chore] Log possible duplicate detectors
* Fix typos
2023-04-18 10:36:00 -05:00
1c89e79c2d
Remove toLower call on decoded chunk ( #1254 )
...
* remove to lower on decoded data
* clean up
2023-04-14 07:29:32 -05:00
0052f60090
Allow for custom verifier ( #1070 )
...
* allow for custom verifier.
* Update engine.
* use custom detectors.
* set cap.
* Update verifiers.
* Remove nil check.
* resolved nit
* handle uppercase values
* updating missing url logs
* adding more descriptive variable names
* updating logs to use correct variables
* Removing toLower for urls
* if else nits
* Adding versioning for github and gitlab
---------
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com >
Co-authored-by: ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com >
2023-03-29 12:26:39 -07:00
f0b6b5d0d9
add a break statement when iterating through keywords ( #1184 )
2023-03-15 16:51:03 -05:00
4777b77ec6
Keyword optimization ( #1144 )
...
* init
* ignore trufflehog binary and added comment
* remove unused keywords in chunk, better comment
* remove keywords from engine struct
2023-03-02 11:32:37 -06:00
dd39848709
Add ability to include and exclude detectors ( #1106 )
...
* Add ability to include and exclude detectors
* Trim space before checking for empty items
* Explicitly check for integer overflow
* Use strconv.ParseInt instead of strconv.Atoi
* Address comments
2023-02-27 16:46:45 -06:00
58e8c1e4ac
[chore] Remove logrus from engine package ( #1085 )
2023-02-09 16:55:19 -06:00
430d5c764c
Rename and export isGitSource ( #1016 )
2023-01-10 12:51:58 -08:00
8b2e1d36cf
Copy metadata for line number aware sources ( #1011 )
...
* Copy metadata for line number aware sources
* Improve style
2023-01-10 09:35:44 -08:00
335ce85ce4
Export line number code ( #962 )
2022-12-06 15:31:15 -08:00
d3b24fa592
Replace plain decoder with utf8 ( #922 )
2022-11-15 09:36:01 -08:00
fe029b1098
[THOG-793] - Return all unverified results ( #856 )
...
* Remove the check to filter and return only a single unverified result.
* Revert "Remove the check to filter and return only a single unverified result."
This reverts commit 494e432803
2022-10-31 09:36:10 -07:00
034ca4fb5b
Add bytes counter to scans ( #876 )
2022-10-27 12:54:22 -07:00
958266ea84
Run chunker in pipeline ( #859 )
...
* Run chunker in pipeline
* Move ChunkSize and PeekSize to source package.
* Use new Chunk and Peek size location
2022-10-24 13:57:27 -07:00
128002885a
Add decoder type to results. ( #835 )
2022-10-06 11:55:07 -07:00
f2f5b6dba8
Fix recover. ( #817 )
2022-09-22 15:47:42 -07:00
92f40c2031
[THOG-709] - Recover from detector panics ( #810 )
2022-09-22 07:01:10 -07:00
fa9479100e
Add common sentry recover library and add into goroutines ( #738 )
...
* Add common sentry recover library and add into goroutines
* fix nits
2022-08-29 11:45:37 -07:00
6fa2171a22
Refactor Engine to wait for workers in a Finish method ( #581 )
...
* Refactor Engine to wait for workers in a Finish method
This should allow the engine to run multiple concurrent scans if
desired before shutting down.
Additionally, this commit refactors some of the printing logic to the
output package.
* Fix tests
2022-05-25 11:35:44 -05:00
6ee5c57968
Fix overlooked nil ( #523 )
...
* Fix overlooked nil
* Use default case
2022-05-04 15:45:12 -07:00
62bb3c57d3
Keep first line number for each chunk ( #520 )
2022-05-04 14:11:10 -07:00
8d2dd624e4
fetcher controls fetch interval, prevent fetching loop
2022-04-03 21:13:39 -07:00
Shreyas Sriram