This branch adds the ability to forcefully remove a runner from GARM.
When the operator wishes to manually remove a runner, the workflow is as
follows:
* Check that the runner exists in GitHub. If it does, attempt to
remove it. An error here indicates that the runner may be processing
a job. In this case, we don't continue and the operator gets immediate
feedback from the API.
* Mark the runner in the database as pending_delete
* Allow the consolidate loop to reap it from the provider and remove it
from the database.
Removing the instance from the provider is async. If the provider errs out,
GARM will keep trying to remove it in perpetuity until the provider succedes.
In situations where the provider is misconfigured, this will never happen, leaving
the instance in a permanent state of pending_delete.
A provider may fail for various reasons. Either credentials have expired, the
API endpoint has changed, the provider is misconfigured or the operator may just
have removed it from the config before cleaning up the runners. While some cases
are recoverable, some are not. We cannot have a situation in which we cannot clean
resources in garm because of a misconfiguration.
This change adds the pending_force_delete instance status. Instances marked with
this status, will be removed from GARM even if the provider reports an error.
The GARM cli has been modified to give new meaning to the --force-remove-runner
option. This option in the CLI is no longer mandatory. Instead, setting it will mark
the runner with the new pending_force_delete status. Omitting it will mark the runner
with the old status of pending_delete.
Fixes: #160
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change adds a new controller info endpoint and associated client and
CLI command. The controller info endpoint returns information about controller
status and configuration.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Updating a pool will no longer try to create a pool manager if one does
not already exist. A pool manager must be started when a pool is created.
Updating an existing pool without a pool manager is an error condition.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change renames the module from "garm" to "github.com/cloudbase/garm".
This will make it easier to consume public functions defined in garm, by
external applications, without having to resort to replace.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Wait for http server graceful shutdown and for pool managers to
properly exit.
* Fix potential nil pointer dereference when checking response
code from github API.
Define a metadata subrouter and move the token endpoint there. We may
end up needing multiple endpoints for various purposes in the future.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change adds a github registration endpoint that instances can use
to fetch a github registration token.
This change also invalidates disables access to an instance to the token
and status updates endpoints once the instance transitions from
"pending" or "installing" to any other state.
Garm no longer fails on startup if a pool manager cannot be started. It
will attempt to start the pool manager in the background. If it fails
due to an unauthorized error, it will sleep for 3 hours. It is unlikely
it will work a second time if credentials are not updated in the config
and garm is restarted, so no point in getting rate limited.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
The GitHub credentials section now allows setting some API endpoints
that point the github client and the runner setup script to the propper
URLs. This allows us to use garm with an on-prem github enterprise server.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
In order to allow mocking for some of the `runner` functions, we created a
separate interface (called `PoolManagerController`) with `Create`, `Get`,
`Delete` operations for the `organization` / `repository` pool managers.
Furthermore, a new runner struct (`poolManagerCtrl`) implements this new
interface. The existing code is refactored to use the `poolManagerCtrl`
whenever the pool managers for `org` / `repo` are handled.
This allows more unit testing for the runner functions since `poolManagerCtrl`
field can be mocked now.
Besides this, there are some typos fixed as well.
* added interface for the github client. This will help mocking it
out for testing.
* removed some unused code
* moved some code around
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Runners can now be manually removed using the CLI. Some restrictions apply:
* A runner must be idle in github. Github will not allow us to remove a runner
that is running a workflow.
* The runner status must be "running"
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
