This change adds the API endpoints, the CLI commands and the web UI elements
needed to manage objects in GARMs internal storage.
This storage system is meant to be used to distribute the garm-agent and as a
single source of truth for provider binaries, when we will add the ability for GARM
to scale out.
Potentially, we can also use this in air gapped systems to distribute the runner binaries
for forges that don't have their own internal storage system (like GHES).
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Use uncompressed tools for gitea. Gitea compresses using .xz, including for
Windows, which does not have a native, built-in tool to uncompress that
format.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
The code that attempts to fetch tools from upstream, must return the error
if the current setting is the upstream repo.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change adds 2 new options to gitea forge endpoints:
* Tools metadata URL
* Use internal tools URLs
By default, GARM looks in the releases page of the gitea arc_runner
to determine where it can download the runner binary from for a particular
OS/arch. The tools metadata URL option can be set on an endpoint and can point
to a mirror of the upstream repo. The requirement is that the asset names
exactly mirror upstream naming conventions.
The second option disables GARM calling out to the tools metadata URL entirely.
GARM has some hardcoded values for nightly binaries. If this option is checked,
GARM will use those values, without making any kind of outgoing API call to
determine availability. This is useful in air-gapped environments.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Add template api endpoints
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Added template bypass
Pools and scale sets will automatically migrate to the new template
system for runner install scripts. If a pool or a scale set cannot be
migrate, it is left alone. It is expected that users set a runner install
template manually for scenarios we don't yet have a template for (windows
on gitea for example).
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Integrate templates with pool create/update
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Add webapp integration with templates
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Add unit tests
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Populate all relevant context fields
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Update dependencies
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Fix lint
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Validate uint
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Add CLI template management
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Some editor improvements and bugfixes
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Fix scale set return values post create
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Fix template websocket events filter
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
---------
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change simplifies the scale down logic a bit. It also make sure we
don't accidentally remove runners that are in the process of being created
when we try to consolidate.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change adds some more cache helper functions, additional tests,
vastly improves memory usage when loading instances and cleans up some
code.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Remove calls to github after creating a runner. It just adds overhead
for very little benefit.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change adds a number of fixes for scale sets:
* Reset last message ID when we need to recreate the scale set in GitHub.
Message ID gets reset in github when this happens and we end up ignoring
messages because we see that they are older than we have recorded.
* Clean up deleted instances from state scale set state
* Properly stop instance handler in the provider worker when an update
operation comes in that signals that an instance has been marked as "deleted"
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
The debug-log command now supports log level filtering and attribute
filtering. The log level filtering will only be able to set the minimum
log level as low as the server is configured to stream. If the server has
its log level set as INFO, then setting the log level in the CLI to DEBUG
will have no effect.
But anything above what the server sends, is within the control of the client
to filter. This is all done client side.
Attribute filters are useful if you need to watch the logs for a particular
worker, entity, etc.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
On github, attempt to use the scaleset API to list all runners without
pagination. This will avoid missing runners and accidentally removing them.
Fall back to paginated API if we can't use the scaleset API.
Add ability to retrieve all instances from cache, for an entity.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Github will remove inactive scale sets after 7 days. This change
ensures the scale set exists in github before spinning up the listener.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change adds a single page application front-end to GARM. It uses
a generated REST client, built from the swagger definitions, the websocket
interface for live updates of entities and eager loading of everything
except runners, as users may have many runners and we don't want to load
hundreds of runners in memory.
Proper pagination should be implemented in the API, in future commits,
to avoid loading lots of elements for no reason.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Filter out gitea tools to only consider archived downloads. This
should help in situations where bandwidth is more important than
CPU time used to unarchive the tools.
Also a drive by fix for scale sets cleanup.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
There seems to be a change in the scale set message. It now includes
a jobID and sets the runner request ID to 0. This change adds separate
job ID fields for workflow jobs and scaleset jobs.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Do not look for a name when composing the scale set. Preload may not
have been called on an entity, but we still have the ID, which is the
only thing needed when GetEntity() is called.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* time.NewTicker will panic if the duration is 0. Make it return
early if duration is 0.
* Return a pre-closed channel in Wait() instead of nil. Ensures receiver
will not block forever.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Use JoinPath() in newActionsRequest() and make sure we pass relative
paths to it. This should fix scale sets on GHES.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change adds the ability to filter the list of entities returned
by the API by entity owner, name or endpoint, depending on the entity
type.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* We were passing the wrong type to GORM for events
* We now expose entity events in the API and CLI
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change renames a lot of variables, types and functions to be more
generic. The goal is to allow GARM to add more forges in the future.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change adds a loop that keeps a cache of credentials rate limits
as reported by the github API. The cache is updated every 30 seconds
and is purely informational for the user.
This change also adds some caching improvements. Functions that return
values from the cache as lists, will now sort by ID or creation date.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This change adds an update routine in the cache worker, for github tools
downloads.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
* Split the main function into a couple of more functions
* Add credentials, entity, pool and scaleset cache
* add credentials worker that updates the cache
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Adds a periodic cleanup function that cross checks runners between github,
the provider and the GARM database. If an inconsistency is found, GARM will
attempt to fix it.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
