c5af979aee
Assume unauthenticated github scans have public visibility ( #2308 )
2024-01-16 14:57:06 -08:00
9408425cc6
[chore] - small updates ( #2288 )
...
* small updates
* fix logic
* simplify fxn
* remove errors
* use strings.EqualFold
2024-01-11 14:27:10 -08:00
677238c96c
Extend memory cache ( #2275 )
...
* Extend memory cache to allow for configuring custom expiration and purge interval
* use any for value type
* fix test
* fix test
* address comments
* address
* make new construct more clear
* reduce duplication
* fix test
2024-01-11 08:20:37 -08:00
7d93adc1d0
Add skip archive support ( #2257 )
2023-12-22 11:55:23 -08:00
64c7365364
add secretID to chunk ( #2242 )
2023-12-18 15:27:49 -08:00
5c6ce693c1
[feat] - Make skipping binaries configurable ( #2226 )
...
* Make skipping binaries configurable
* remove ioutil
* fix
* address comments
* address comments
* use multi-reader
* remove print
* use const
* fix test
* fix my stupidness
2023-12-15 11:46:27 -08:00
b3040b1227
fix(github): remove unused 'members' var ( #2202 )
2023-12-14 11:53:24 -08:00
b75991850a
[chore] - Compile regex once ( #2176 )
...
* move regex compilation out of the fxn
* missed a spot
* merge main
2023-12-07 07:26:27 -08:00
024aa056b9
chore(github): add a newline between titles and bodies ( #2124 )
2023-11-23 16:14:28 -08:00
1f502fd42c
feat(github): scan issue & pr titles ( #1899 )
2023-11-22 19:15:27 -08:00
52600a897a
[chore] Replace chunks channel with ChunkReporter in git based sources ( #2082 )
...
ChunkReporter is more flexible and will allow code reuse for unit
chunking. ChanReporter was added as a way to maintain the original
channel functionality, so this PR should not alter existing behavior.
2023-11-01 09:22:44 -07:00
b8724e87e6
Use the configured include repositories in the GitHub filter ( #1926 )
2023-10-20 19:03:28 -07:00
3acc65b2fb
chore(github): reduce comment log verbosity ( #1922 )
2023-10-20 16:16:38 -07:00
7ac7fa8728
Move Github comments check to fix a test #1927
2023-10-19 19:23:55 -04:00
4b821e9732
Handle secondary GitHub ratelimits ( #1912 )
...
* fix(github): reduce visibility-related api calls
* fix(github): handle secondary ratelimits
2023-10-19 14:54:45 -04:00
6ea3a7da4a
fix(github): normalize repo cache ( #1897 )
2023-10-17 15:07:47 -07:00
3d2490ca80
use Repositories field from conn. ( #1860 )
2023-10-04 13:56:02 -07:00
699547b7d3
consolidated pr and issue descr/comment flags ( #1827 )
2023-09-27 15:54:02 -04:00
bf47fd69bb
Github partial scan ( #1804 )
...
* Add ability for targetted partial scans of Github.
* update comment.
* add more tests.
* add additiional test.
* address comments.
2023-09-26 12:38:33 -07:00
1e42dae734
added PR and Issue body scanning ( #1816 )
...
* added PR and Issue body scanning; adjusted CLI args to fit
* removed print statement from debugging
* removed exclude-commits; adjusted CLI flags
* minor changes to match main branch
* fixing logic
* updating README for --issues and --prs
2023-09-26 12:25:48 -04:00
dbcb888063
Update Source interface to use SourceID and JobID types ( #1774 )
...
The previous implementation used int64 for both, which can be mixed up
easily. Using distinct types adds a layer of type safety checked by the
compiler.
2023-09-14 11:28:24 -07:00
72b6a9ec6b
Add a SourceType constant to all source packages ( #1768 )
2023-09-12 17:23:25 -07:00
2a9f34962d
Add optional param to Chunks ( #1747 )
...
* Add interface for targeted chunking.
* use optional args.
* update Chunks method signature.
* update tests.
* fix test.
* update QueryCriteria type.
2023-09-07 09:03:37 -07:00
2b1b1b5ad0
Add jobID to chunk. ( #1721 )
2023-08-29 12:02:30 -07:00
0932ea224b
[chore] - Prevent nil deref panic ( #1709 )
2023-08-26 20:39:50 -07:00
4f4a79f62b
Support azure git links ( #1662 )
...
* Support azure git links.
* update comment.
* update test names.
2023-08-24 14:36:52 -07:00
a2a7a2087e
[chore] - update comments and logs. ( #1654 )
...
* update comments and logs.
* Update github.go
2023-08-23 13:18:07 -07:00
fd00d2b30b
add rate limit and consumption metrics for GitHub ( #1651 )
...
* add rate limit and consumption metrics
* incrment after each repo scanned
* update repo scanned label name
2023-08-22 15:01:59 -04:00
9a13c74a35
add thog CLI support for GitHub config validate ( #1626 )
...
* add exportable validate function for github
* update validator
* use the context
* gate to prevent panic
* wrap error with context
* wrap error with context for basic auth and unauth
2023-08-22 10:22:39 -04:00
d51e3b6d83
Only scan gist comments or repo comments. ( #1646 )
2023-08-20 11:38:28 -07:00
0ae8cf5d35
[bug] - handle IOOR panic ( #1639 )
...
* handle IOOR panic.
* use a better fxn name.
* increae timeout for test to compete.
* simplify code and add test.
* do it for miccah.
2023-08-17 15:47:11 -07:00
db89e345d7
correct logging output for github comments and add oss flags ( #1632 )
...
* correct logging output
* add flags
* respect oss cli flags for github comment scanning
* improve copy
2023-08-16 18:23:59 -04:00
eb00d0d4e1
[bug] - fix data races ( #1577 )
...
* fix data race.
* Add test and fix additional data race.
* address comments.
2023-07-31 11:12:38 -07:00
ad57de50cd
Do not nest transports for Github installation client ( #1564 )
...
#1454 modified one of the Github enumeration code paths in a way that broke an integration test by causing one client's transport to be used for the construction of a different client, causing authentication failures. This saves the original transport for use, fixing the test.
2023-07-31 11:31:16 -04:00
e0faac8d1c
Fix runtime error when scanning Gist comments ( #1552 )
...
* fix(github): fix runtime error from gist comments
* fix(github): add flag to scan Gist comments
2023-07-31 08:57:42 -05:00
46823f77c9
feat(github): clarify comment log statement ( #1553 )
2023-07-26 09:40:30 -05:00
2290954b02
fix(github): use apiEndpoint for basic or no auth ( #1454 )
2023-07-25 20:03:08 -07:00
1594fddf05
feat(git): include line in github & gitlab links ( #1466 )
2023-07-11 20:02:27 -07:00
4334af4d34
scan GitHub PR and issue comments ( #1435 )
...
* issue comment scanning
* save progress
* test
* test for pr comment and issue comment
* add pagination support
* linter stuff
* make linter happy
* remove debug log
* readd logging
* github issue resolved
* var const block and handle rate limit
* remove magic number
* make gitURLParse a public function to use more generally
* fix test bug
* make comment scanning OPT-IN
2023-07-11 15:13:33 -04:00
f3152b6885
Implement SourceUnitUnmarshaller for all sources ( #1416 )
...
* Implement CommonSourceUnitUnmarshaller
* Add SourceUnitUnmarshaller to all sources using
All sources, with the exception of git, will use the CommonSourceUnit as
they only contain a single type of unit to scan.
* Fix method comments to adhere to Go's style guide
2023-06-23 11:15:51 -05:00
c8944825de
Surface missing git as an error during initialization ( #1362 )
2023-05-26 15:23:08 -07:00
1da7720912
Replace context.TODO. ( #1349 )
2023-05-19 11:09:51 -07:00
31844b12e3
[oc-313] - Add GitHub metrics ( #1324 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Use separate file for repo logic.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
* Add logs for duration and repo size.
* fix integration test.
* address comment.
2023-05-16 08:45:28 -07:00
4250773e92
GitHub basic auth ( #1337 )
2023-05-15 22:04:42 -07:00
030c093392
Fix how we scan orgs ( #1327 )
...
* Fix how we scan orgs.
* fix integration test.
2023-05-04 08:07:11 -07:00
323c093818
Normalize GitHub repos during enumeration ( #1269 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
2023-05-03 08:35:53 -07:00
a2266b4e28
add additional logging ( #1298 )
...
* add additional logging.
* update test.
* remove continue.
* address comments.
2023-04-27 16:48:04 -07:00
c6826c4574
Fix nil scan options ( #1107 )
2023-02-14 12:09:45 -06:00
f2139a7615
Github filter support for exclude and include ( #1087 )
...
* test
* Add missing head and base hash back.
---------
Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com >
2023-02-14 08:40:53 -08:00
0ce72ccda3
[chore] Remove logrus from github source ( #1086 )
...
* [chore] Remove logrus from github source
* Fix handleRateLimit test
* Fix tests
2023-02-09 18:02:04 -06:00
Miccah