Commit graph

95 commits

Author SHA1 Message Date
Michael Kuhnt
6cd18ff1fd
improve access to controller info 2023-01-26 22:13:18 +01:00
Michael Kuhnt
6a032bfaa2
metrics: fix review findings 2023-01-26 15:46:27 +01:00
Michael Kuhnt
ee659f509f
feat: add prometheus metrics & endpoint 2023-01-26 14:15:16 +01:00
mgoeppe
dce1808860 fixed runner tests 2023-01-24 08:51:25 +01:00
mgoeppe
f9f917ba05 aligned code enterprises,organizations and repositories and fixed sql tests 2023-01-24 08:51:25 +01:00
Gabriel Adrian Samfira
e93b6d73e5
Sanitize log entries
While most of these log entries come from either github or our own
database, it's still a good idea to sanitize them.
2023-01-23 18:01:46 +02:00
Gabriel Adrian Samfira
b354cedf7e
Fixed a bunch of linting issues
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-01-20 22:21:22 +02:00
Gabriel Adrian Samfira
f2cf947c00
Move pool type in params
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-01-20 17:08:15 +02:00
Gabriel Adrian Samfira
abcc9569bd
Add a common RunnerPrefix type
There are several fields that are common among some of the data
structures in garm. The RunnerPrefix is just one of them. Perhaps we
should move some of the rest in a common type and embed that into the
types that share those fields.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-01-20 12:12:15 +02:00
Michael Kuhnt
49762d7f9e
fix: shortid generates correctly now 2023-01-19 14:23:40 +01:00
Michael Kuhnt
6af3025743
feat: allow to configure the runner name 2023-01-19 11:13:36 +01:00
Michael Kuhnt
3a46a9d127
feat: scale down idle runners 2023-01-10 18:18:28 +01:00
Gabriel Adrian Samfira
b954038624 Ensure loop closes properly and provider update
* Ensure the pool loop exits properly when the pool is not yet in
a running state.
  * Use ListInstances() when cleaning orphaned runners. This ensures
We only run one API call per pool to list instances, instead of running
a GetInstance() for each individual instance we are checking.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-01-08 16:40:42 +00:00
Gabriel Adrian Samfira
d5f5524934 Wait for loop exit and some fixes
* Wait for http server graceful shutdown and for pool managers to
properly exit.
  * Fix potential nil pointer dereference when checking response
code from github API.
2022-12-30 15:07:40 +00:00
Gabriel Adrian Samfira
6fc950e513 Readd the instance status check
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-12-29 23:45:17 +00:00
Gabriel Adrian Samfira
a91f64331e Limit instances to one runner token 2022-12-29 22:57:10 +00:00
Gabriel Adrian Samfira
3a92a5be0e Some cleanup and safety checks
* Add logging middleware
  * Remove some noise from logs
  * Add some safety checks when managing runners
2022-12-29 16:50:11 +00:00
Gabriel Adrian Samfira
2085ad6945 Remove unused field and sync azure provider
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-12-06 20:01:10 +00:00
Gabriel Adrian Samfira
d3fe741cfe Don't save runner registration tolen in DB
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-12-06 19:48:00 +00:00
Gabriel Adrian Samfira
0b50397b47 Make sure to decode token
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-12-06 15:28:36 +00:00
Gabriel Adrian Samfira
cb5baeb547 Add Enterprise tests 2022-12-04 17:30:42 +00:00
Gabriel Adrian Samfira
0869073906 Define a metadata subrouter
Define a metadata subrouter and move the token endpoint there. We may
end up needing multiple endpoints for various purposes in the future.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-12-02 19:48:38 +00:00
Gabriel Adrian Samfira
a078645ab2
Add token endpoint
This change adds a github registration endpoint that instances can use
to fetch a github registration token.

This change also invalidates disables access to an instance to the token
and status updates endpoints once the instance transitions from
"pending" or "installing" to any other state.
2022-12-01 18:00:22 +02:00
Gabriel Adrian Samfira
970ffb608e
Mark pool manager as offline in case of 403
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-10-21 17:14:03 +03:00
Gabriel Adrian Samfira
05057e37fd
Start pool managers in the background
Garm no longer fails on startup if a pool manager cannot be started. It
will attempt to start the pool manager in the background. If it fails
due to an unauthorized error, it will sleep for 3 hours. It is unlikely
it will work a second time if credentials are not updated in the config
and garm is restarted, so no point in getting rate limited.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-10-21 17:14:03 +03:00
Gabriel Adrian Samfira
80452aac39
Update go-github and remove redirect 2022-10-21 17:14:03 +03:00
Gabriel Adrian Samfira
adaeaa48f8
Properly paginate
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-10-21 17:14:03 +03:00
Gabriel Adrian Samfira
b8e3303e54
Lower tools update interval 2022-10-21 17:14:03 +03:00
Gabriel Adrian Samfira
afb2bd9ab8
Fix internal config credentials
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-10-21 17:14:03 +03:00
Gabriel Adrian Samfira
3e3b91ee59
Add enterprise support to garm-cli
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-10-21 17:14:03 +03:00
Gabriel Adrian Samfira
296333412a
Add enterprise support
This change adds enterprise support throughout garm.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-10-21 17:14:03 +03:00
Gabriel Adrian Samfira
f40420bfb6
Add ability to specify github enpoints for creds
The GitHub credentials section now allows setting some API endpoints
that point the github client and the runner setup script to the propper
URLs. This allows us to use garm with an on-prem github enterprise server.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-10-21 17:14:03 +03:00
Gabriel Adrian Samfira
a7f151e2d2
Add log streamer
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-10-21 11:13:42 +03:00
Gabriel Adrian Samfira
49fbc110d5
Don't remove foreign runners
When garm starts up, it attempts to cleanup any orphaned runners from
github. This, unfortunately ends up deleting manually set up runners.
This change looks for our controller ID in the labels of the runners and
only deleted runners that we created.

Fixes: #40

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-10-17 18:45:48 +03:00
Gabriel Adrian Samfira
0ff2fd2e06
Make owners and repos case insensitive
Github treats owners (users and orgs) and repos as case insensitive. To
github, https://github.com/cloudbase/garm is equivalent to
https://github.com/CloudBase/GaRm. This commit makes the sql store
backend, case insensitive when querying repos and orgs.

Fixes: #25

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-10-05 18:14:07 +03:00
Gabriel Adrian Samfira
e8fa6dba6e
Attempt to fetch runner name from API
In some cases, runner information is not sent via webhook by Github when
a workflow job transitions to in_progress. We need to know the runner
name in order to update the state in the database. Attempt to fetch the
runner from the API using the workflow ID.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-09-20 14:46:44 +03:00
mihaelabalutoiu
e2ada6d1e3 Cleanup unused struct field
Remove `Pool` field from `CreateInstanceParams` struct, because
this is given as a separate parameter to the `CreateInstance` function.
2022-09-15 19:11:37 +03:00
Gabriel Adrian Samfira
b4ac128673
Create a testing package for common utilities
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-09-15 17:48:09 +03:00
mihaelabalutoiu
8c910d904f Cleanup duplicate code 2022-09-14 12:55:26 +03:00
Gabriel
5759b74e0d
Merge pull request #12 from mihaelabalutoiu/add-repositories-unit-tests
Add `repositories.go` unit tests
2022-09-09 17:51:11 +03:00
mihaelabalutoiu
8215edfe26 Update organization.go unit tests 2022-09-09 16:38:52 +03:00
mihaelabalutoiu
54f2dd6154 Add repositories.go unit tests 2022-09-09 15:57:18 +03:00
mihaelabalutoiu
c5158fbf5c Add more repositories.go unit testing 2022-09-09 10:47:36 +03:00
mihaelabalutoiu
505489eddf Add repositories.go unit tests 2022-09-07 16:49:42 +03:00
mihaelabalutoiu
1a7c31c8ea Add more organizations.go unit testing
Also, update vendor with the `github.com/stretchr/testify/suite`.
2022-09-06 18:04:51 +03:00
mihaelabalutoiu
3094ad09be Add PoolManagerController mocks 2022-09-01 16:08:56 +03:00
Ionut Balutoiu
7b6c2e6106 Refactor code to allow more unit testing
In order to allow mocking for some of the `runner` functions, we created a
separate interface (called `PoolManagerController`) with `Create`, `Get`,
`Delete` operations for the `organization` / `repository` pool managers.

Furthermore, a new runner struct (`poolManagerCtrl`) implements this new
interface. The existing code is refactored to use the `poolManagerCtrl`
whenever the pool managers for `org` / `repo` are handled.

This allows more unit testing for the runner functions since `poolManagerCtrl`
field can be mocked now.

Besides this, there are some typos fixed as well.
2022-08-18 17:47:05 +03:00
mihaelabalutoiu
a869b5d743 Add unit tests 2022-08-01 12:34:59 +03:00
Gabriel Adrian Samfira
067197c1b5 Fix instance JWT token expiration
The instance JWT token expiration time was set at 15 minutes, regardless
of bootstrap timeout. This meant that instances that take longer than 15
minutes, would not be able to send their status updates and github agent
ID back to garm.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-07-17 07:24:19 +00:00
Gabriel Adrian Samfira
3da860eea6 Always lookup images in simplestreams
This change makes sure we always download the latest image from
simplestreams. We no longer copy labels from simplestreams so we need
to lookup the hash of the image against simplestreams. This means that
in airgapped environments, the operator will have to manually tag images
and use the tag name in favor of remote:image_name.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2022-07-16 13:47:17 +00:00