7ac7fa8728
Move Github comments check to fix a test #1927
2023-10-19 19:23:55 -04:00
4b821e9732
Handle secondary GitHub ratelimits ( #1912 )
...
* fix(github): reduce visibility-related api calls
* fix(github): handle secondary ratelimits
2023-10-19 14:54:45 -04:00
6ea3a7da4a
fix(github): normalize repo cache ( #1897 )
2023-10-17 15:07:47 -07:00
3d2490ca80
use Repositories field from conn. ( #1860 )
2023-10-04 13:56:02 -07:00
699547b7d3
consolidated pr and issue descr/comment flags ( #1827 )
2023-09-27 15:54:02 -04:00
bf47fd69bb
Github partial scan ( #1804 )
...
* Add ability for targetted partial scans of Github.
* update comment.
* add more tests.
* add additiional test.
* address comments.
2023-09-26 12:38:33 -07:00
1e42dae734
added PR and Issue body scanning ( #1816 )
...
* added PR and Issue body scanning; adjusted CLI args to fit
* removed print statement from debugging
* removed exclude-commits; adjusted CLI flags
* minor changes to match main branch
* fixing logic
* updating README for --issues and --prs
2023-09-26 12:25:48 -04:00
dbcb888063
Update Source interface to use SourceID and JobID types ( #1774 )
...
The previous implementation used int64 for both, which can be mixed up
easily. Using distinct types adds a layer of type safety checked by the
compiler.
2023-09-14 11:28:24 -07:00
72b6a9ec6b
Add a SourceType constant to all source packages ( #1768 )
2023-09-12 17:23:25 -07:00
2a9f34962d
Add optional param to Chunks ( #1747 )
...
* Add interface for targeted chunking.
* use optional args.
* update Chunks method signature.
* update tests.
* fix test.
* update QueryCriteria type.
2023-09-07 09:03:37 -07:00
2b1b1b5ad0
Add jobID to chunk. ( #1721 )
2023-08-29 12:02:30 -07:00
0932ea224b
[chore] - Prevent nil deref panic ( #1709 )
2023-08-26 20:39:50 -07:00
4f4a79f62b
Support azure git links ( #1662 )
...
* Support azure git links.
* update comment.
* update test names.
2023-08-24 14:36:52 -07:00
a2a7a2087e
[chore] - update comments and logs. ( #1654 )
...
* update comments and logs.
* Update github.go
2023-08-23 13:18:07 -07:00
fd00d2b30b
add rate limit and consumption metrics for GitHub ( #1651 )
...
* add rate limit and consumption metrics
* incrment after each repo scanned
* update repo scanned label name
2023-08-22 15:01:59 -04:00
9a13c74a35
add thog CLI support for GitHub config validate ( #1626 )
...
* add exportable validate function for github
* update validator
* use the context
* gate to prevent panic
* wrap error with context
* wrap error with context for basic auth and unauth
2023-08-22 10:22:39 -04:00
d51e3b6d83
Only scan gist comments or repo comments. ( #1646 )
2023-08-20 11:38:28 -07:00
0ae8cf5d35
[bug] - handle IOOR panic ( #1639 )
...
* handle IOOR panic.
* use a better fxn name.
* increae timeout for test to compete.
* simplify code and add test.
* do it for miccah.
2023-08-17 15:47:11 -07:00
db89e345d7
correct logging output for github comments and add oss flags ( #1632 )
...
* correct logging output
* add flags
* respect oss cli flags for github comment scanning
* improve copy
2023-08-16 18:23:59 -04:00
eb00d0d4e1
[bug] - fix data races ( #1577 )
...
* fix data race.
* Add test and fix additional data race.
* address comments.
2023-07-31 11:12:38 -07:00
ad57de50cd
Do not nest transports for Github installation client ( #1564 )
...
#1454 modified one of the Github enumeration code paths in a way that broke an integration test by causing one client's transport to be used for the construction of a different client, causing authentication failures. This saves the original transport for use, fixing the test.
2023-07-31 11:31:16 -04:00
e0faac8d1c
Fix runtime error when scanning Gist comments ( #1552 )
...
* fix(github): fix runtime error from gist comments
* fix(github): add flag to scan Gist comments
2023-07-31 08:57:42 -05:00
46823f77c9
feat(github): clarify comment log statement ( #1553 )
2023-07-26 09:40:30 -05:00
2290954b02
fix(github): use apiEndpoint for basic or no auth ( #1454 )
2023-07-25 20:03:08 -07:00
1594fddf05
feat(git): include line in github & gitlab links ( #1466 )
2023-07-11 20:02:27 -07:00
4334af4d34
scan GitHub PR and issue comments ( #1435 )
...
* issue comment scanning
* save progress
* test
* test for pr comment and issue comment
* add pagination support
* linter stuff
* make linter happy
* remove debug log
* readd logging
* github issue resolved
* var const block and handle rate limit
* remove magic number
* make gitURLParse a public function to use more generally
* fix test bug
* make comment scanning OPT-IN
2023-07-11 15:13:33 -04:00
f3152b6885
Implement SourceUnitUnmarshaller for all sources ( #1416 )
...
* Implement CommonSourceUnitUnmarshaller
* Add SourceUnitUnmarshaller to all sources using
All sources, with the exception of git, will use the CommonSourceUnit as
they only contain a single type of unit to scan.
* Fix method comments to adhere to Go's style guide
2023-06-23 11:15:51 -05:00
c8944825de
Surface missing git as an error during initialization ( #1362 )
2023-05-26 15:23:08 -07:00
1da7720912
Replace context.TODO. ( #1349 )
2023-05-19 11:09:51 -07:00
31844b12e3
[oc-313] - Add GitHub metrics ( #1324 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Use separate file for repo logic.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
* Add logs for duration and repo size.
* fix integration test.
* address comment.
2023-05-16 08:45:28 -07:00
4250773e92
GitHub basic auth ( #1337 )
2023-05-15 22:04:42 -07:00
030c093392
Fix how we scan orgs ( #1327 )
...
* Fix how we scan orgs.
* fix integration test.
2023-05-04 08:07:11 -07:00
323c093818
Normalize GitHub repos during enumeration ( #1269 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
2023-05-03 08:35:53 -07:00
a2266b4e28
add additional logging ( #1298 )
...
* add additional logging.
* update test.
* remove continue.
* address comments.
2023-04-27 16:48:04 -07:00
c6826c4574
Fix nil scan options ( #1107 )
2023-02-14 12:09:45 -06:00
f2139a7615
Github filter support for exclude and include ( #1087 )
...
* test
* Add missing head and base hash back.
---------
Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com >
2023-02-14 08:40:53 -08:00
0ce72ccda3
[chore] Remove logrus from github source ( #1086 )
...
* [chore] Remove logrus from github source
* Fix handleRateLimit test
* Fix tests
2023-02-09 18:02:04 -06:00
4ef546a06b
fix github integration tests ( #1042 )
2023-01-25 08:57:39 -08:00
d720c0c0f3
Switch to retryableHttpClient for GitHub AuthN API Client + More Logs ( #995 )
...
* Adding missing flags to Readme
* Use retryableHttpClient by default for GitHub
* Adding repoUrl for scanning time log
* Use WithField instead of WithFields
* Updating README with lasted --help output
2023-01-09 09:21:56 -08:00
009756dce6
add proto that was missing. ( #986 )
2022-12-23 13:27:07 -08:00
936a139596
Allow using a glob for include list. ( #977 )
...
* Allow using a glob for include list.
* Update command flag.
* Make comment more clear.
* update comment.
* Allow scanning repo and org at the same time.
2022-12-16 13:28:16 -08:00
7de9bdd12d
Support globbing with ignore repos ( #967 )
2022-12-09 12:10:42 -08:00
a72b9feb35
Only scan org with --org flag. ( #931 )
2022-12-06 16:18:48 -08:00
dd141fb55f
[oc-147] - Add context to all git methods ( #901 )
...
* Add context to all git methods.
* remove logrus.
* Add ctx.
* Address comments.
* Add error to clone failing.
* Return error.
2022-11-03 16:36:52 -07:00
fe1e475a04
Prevent concurrent read and writes to visibility map. ( #892 )
2022-11-01 16:20:59 -07:00
fb56b9f713
Check rate limit when getting github user ( #855 )
...
Also, don't fetch a github user or their token when both are known. This
currently only affects the Github Token auth type. Github App
installations will continually fetch tokens every time we clone a repo.
In the future we should check the `ExpiresAt` field of the Github App
token and determine if we need to fetch a new one at that point.
2022-10-20 18:14:28 -04:00
029519eb01
[THOG-767] ignore gitlab repos ( #853 )
...
* Add ability to ignore repos.
* use std library slices.Contains.
* Add tests.
* Remove zero values from test.
2022-10-19 13:55:44 -07:00
2d6aadcb46
[THOG-774] - GitHub ignore repo full name ( #848 )
...
* Use github repo full name.
* fix tests.
2022-10-14 09:20:49 -07:00
04c9bb535e
[THOG-768] - Add ability to skip scanning Github repos ( #846 )
...
* Add ability to skip scanning Github repos.
* remove old change.
* rename method.
2022-10-12 16:28:24 -07:00
891996f546
Do not fail scanning if we cannot enumerate gists ( #826 )
2022-09-27 20:59:10 +02:00
Cody Rose