by adding the context from main and make auth.GetAdminContext accepting a
context we are now able to stop the metrics collection loop once the
context is canceled
Signed-off-by: Mario Constanti <mario.constanti@mercedes-benz.com>
refactoring is needed to make the metrics package usable from within the
runner package for further metrics.
This change also makes the metric-collector independent from requests to
the /metrics endpoint
Signed-off-by: Mario Constanti <mario.constanti@mercedes-benz.com>
This change increases the tools refresh interval to 5 minutes, cleans
up the websocket code a bit, augments the error message that may be returned
when trying to delete a runner in an invalid state and removes a log message
that does not add much value.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Pool managers will have 2 fields identifying which manager generated
the log line.
In the future, we will add tracking ids in various cases, allowing
us to track down issues faster.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change switches GARM to the new structured logging standard
library. This will allow us to set log levels and reduce some of
the log spam.
Given that we introduced new knobs to tweak logging, the number of
config options for logging now warrants it's own section.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Allow runners to update their own system information. Runners can now send
back os_name, os_version and agent_id back as part of a POST to
CALLBACK_URL/system-info/.
The goal is to get better info in regard to the actual OS that's running
and to move the agent_id from the status updates to the system-info callback.
The status updates should be used only to send back info about the status of
the installation process.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Canonical have relicensed the LXD project to AGPLv3. This means that we can
no longer update the go LXD client without re-licensing GARM as AGPLv3. This
is not desirable or possible.
The existing code seems to be Apache 2.0 and all code that has already been
contributed seems to stay as Apache 2.0, but new contributions from Canonical
employees will be AGPLv3.
We cannot risc including AGPLv3 code now or in the future, so we will separate
the LXD provider into its own project which can be AGPLv3. GARM will simply
execute the external provider.
If the client code of LXD will ever be split from the main project and re-licensed
as Apache 2.0 or a compatible license, we will reconsider adding it back as a
native provider. Although in the long run, I believe external providers will
be the only option as they are easier to write, easier to maintain and safer to
ship (a bug in the provider does not crash GARM itself).
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
On large deployments with many jobs, we cannot check each job that
we recorded in the DB against the GH API.
Before this change, if a job was updated more than 10 minutes ago,
garm would check against the GH api if that job still existed. While
this approach allowed us to maintain a consistent view over which jobs
still exist and which are stale, it had the potential of spamming the
GH API, leading to rate limiting.
This change uses the scale-down loop as an indicator for job staleness.
If a job remains in queued state in our DB, but has dissapeared from GH
or was serviced by another runner and we never got the hook (garm was down
or GH had an issue - happened in the past), then garm will spin up a new
runner for it. If that runner or any other runner is scaled down, we check
if we have jobs in the queue that should have matched that runner. If we did,
there is a high chance that the job no longer exists in GH and we can remove
the job from the queue.
Of course, there is a chance that GH is having issues and the job is never
pushed to the runner, but we can't really account for everything. In this case
I'd rather avoid rate limiting ourselves.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change adds a flag on providers that allows users to disable JIT
configuration even when it's available. For context, JIT is available
on github.com and any GHES instance >=3.10.
This option is a stopgap measure for providers that have not yet been
updated to use JIT configs instead of runner registration tokens.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
as some provider binaries probably need additional environment variables
set (e.g kubernetes as client-go depends on KUBERNETES_SERVICE_ vars) it
should be possible to define a list of environment variables which
should get bypassed into the provider binary execution
This branch adds the ability to forcefully remove a runner from GARM.
When the operator wishes to manually remove a runner, the workflow is as
follows:
* Check that the runner exists in GitHub. If it does, attempt to
remove it. An error here indicates that the runner may be processing
a job. In this case, we don't continue and the operator gets immediate
feedback from the API.
* Mark the runner in the database as pending_delete
* Allow the consolidate loop to reap it from the provider and remove it
from the database.
Removing the instance from the provider is async. If the provider errs out,
GARM will keep trying to remove it in perpetuity until the provider succedes.
In situations where the provider is misconfigured, this will never happen, leaving
the instance in a permanent state of pending_delete.
A provider may fail for various reasons. Either credentials have expired, the
API endpoint has changed, the provider is misconfigured or the operator may just
have removed it from the config before cleaning up the runners. While some cases
are recoverable, some are not. We cannot have a situation in which we cannot clean
resources in garm because of a misconfiguration.
This change adds the pending_force_delete instance status. Instances marked with
this status, will be removed from GARM even if the provider reports an error.
The GARM cli has been modified to give new meaning to the --force-remove-runner
option. This option in the CLI is no longer mandatory. Instead, setting it will mark
the runner with the new pending_force_delete status. Omitting it will mark the runner
with the old status of pending_delete.
Fixes: #160
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This update pulls in the latest version of garm-provider-common which removes
its dependency on go-github, making future updates much less painful.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
We must create the DB entry for a runner with a JIT config included. Adding it later
via an update runs the risk of having the consolidate loop pick up the incomplete instance.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
When using JIT runners, we register the runner on GitHub before we get
a chance to spin up the instance in the provider. In such cases, we end
up with a runner in "offline" state while we're creating the actual resource
that will embody the runner. This change will give runners a chance to come
online before garm tries to clean them up.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
We need to abstract away the tools struct and not have garm-provider-common
depend on go-github just for that one struct. It makes it hard to update
go-github without updating garm-provider-common first and then all the rest
of the providers.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Thic change adds a metadata endpoint that returns a list of root CA
certificates a runner must install in order to be able to validate all
relevant API endpoints it may require. This includes any GHES API that
runs on a self signed certificate.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Updates the garm-provider-common and go-github packages.
* Update sqlToParamsInstance to return an error when unmarshaling
This change is needed to pull in the new Seal/Unseal functions in common.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
If we fail to get the tools for one pool, garm fails to start due to pool
manager startup timeout. Launch the initial tools update function as a
goroutine and return from Start(). If it fails, it will retry, and we won't
block garm from starting.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
