DevFW-CICD/garm
15
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

doc: fixing grammar and mdlint

Browse source
This commit is contained in:
Ionut Cadariu 2023-01-20 20:20:56 +01:00
parent 4ff7df8f6f
commit b95e8a394f
No known key found for this signature in database
GPG key ID: 44DB7090BAA9CE6C
7 changed files with 592 additions and 599 deletions
Download patch file Download diff file Expand all files Collapse all files

102
README.md
View file

@ -1,4 +1,4 @@
# GitHub Actions Runners Manager (garm) # GitHub Actions Runners Manager (garm)
[![Go Tests](https://github.com/cloudbase/garm/actions/workflows/go-tests.yml/badge.svg)](https://github.com/cloudbase/garm/actions/workflows/go-tests.yml) [![Go Tests](https://github.com/cloudbase/garm/actions/workflows/go-tests.yml/badge.svg)](https://github.com/cloudbase/garm/actions/workflows/go-tests.yml)
@ -6,7 +6,7 @@ Welcome to garm!
Garm enables you to create and automatically maintain pools of [self-hosted GitHub runners](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners), with autoscaling that can be used inside your github workflow runs. Garm enables you to create and automatically maintain pools of [self-hosted GitHub runners](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners), with autoscaling that can be used inside your github workflow runs.
The goal of ```garm``` is to be simple to set up, simple to configure and simple to use. It is a single binary that can run on any GNU/Linux machine without any other requirements other than the providers it creates the runners in. It is intended to be easy to deploy in any environment and can create runners in any system you can write a provider for. There is no complicated setup process and no extremely complex concepts to understant. Once set up, it's meant to stay out of your way. The goal of ```garm``` is to be simple to set up, simple to configure and simple to use. It is a single binary that can run on any GNU/Linux machine without any other requirements other than the providers it creates the runners in. It is intended to be easy to deploy in any environment and can create runners in any system you can write a provider for. There is no complicated setup process and no extremely complex concepts to understand. Once set up, it's meant to stay out of your way.
Garm supports creating pools on either GitHub itself or on your own deployment of [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@3.5/admin/overview/about-github-enterprise-server). For instructions on how to use ```garm``` with GHE, see the [credentials](/doc/github_credentials.md) section of the documentation. Garm supports creating pools on either GitHub itself or on your own deployment of [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@3.5/admin/overview/about-github-enterprise-server). For instructions on how to use ```garm``` with GHE, see the [credentials](/doc/github_credentials.md) section of the documentation.
@ -16,18 +16,19 @@ Garm supports creating pools on either GitHub itself or on your own deployment o
You need to have Go installed, then run: You need to have Go installed, then run:
```bash ```bash
git clone https://github.com/cloudbase/garm git clone https://github.com/cloudbase/garm
cd garm cd garm
go install ./... go install ./...
``` ```
You should now have both ```garm``` and ```garm-cli``` in your ```$GOPATH/bin``` folder. You should now have both ```garm``` and ```garm-cli``` in your ```$GOPATH/bin``` folder.
If you have docker/podman installed, you can also build statically linked binaries by running: If you have docker/podman installed, you can also build statically linked binaries by running:
```bash ```bash
make make
``` ```
The ```garm``` and ```garm-cli``` binaries will be built and copied to the ```bin/``` folder in your current working directory. The ```garm``` and ```garm-cli``` binaries will be built and copied to the ```bin/``` folder in your current working directory.
@ -35,69 +36,69 @@ The ```garm``` and ```garm-cli``` binaries will be built and copied to the ```bi
Add a new system user: Add a new system user:
```bash ```bash
useradd --shell /usr/bin/false \ useradd --shell /usr/bin/false \
--system \ --system \
--groups lxd \ --groups lxd \
--no-create-home garm --no-create-home garm
``` ```
The ```lxd``` group is only needed if you have a local LXD install and want to connect to the unix socket to use it. If you're connecting to a remote LXD server over TCP, you can skip adding the ```garm``` user to the ```lxd``` group. The ```lxd``` group is only needed if you have a local LXD install and want to connect to the unix socket to use it. If you're connecting to a remote LXD server over TCP, you can skip adding the ```garm``` user to the ```lxd``` group.
Copy the binary to somewhere in the system ```$PATH```: Copy the binary to somewhere in the system ```$PATH```:
```bash ```bash
sudo cp $(go env GOPATH)/bin/garm /usr/local/bin/garm sudo cp $(go env GOPATH)/bin/garm /usr/local/bin/garm
``` ```
Or if you built garm using ```make```: Or if you built garm using ```make```:
```bash ```bash
sudo cp ./bin/garm /usr/local/bin/garm sudo cp ./bin/garm /usr/local/bin/garm
``` ```
Create the config folder: Create the config folder:
```bash ```bash
sudo mkdir -p /etc/garm sudo mkdir -p /etc/garm
``` ```
Copy the config template: Copy the config template:
```bash ```bash
sudo cp ./testdata/config.toml /etc/garm/ sudo cp ./testdata/config.toml /etc/garm/
``` ```
Copy the external provider (optional): Copy the external provider (optional):
```bash ```bash
sudo cp -a ./contrib/providers.d /etc/garm/ sudo cp -a ./contrib/providers.d /etc/garm/
``` ```
Copy the systemd service file: Copy the systemd service file:
```bash ```bash
sudo cp ./contrib/garm.service /etc/systemd/system/ sudo cp ./contrib/garm.service /etc/systemd/system/
``` ```
Change permissions on config folder: Change permissions on config folder:
```bash ```bash
sudo chown -R garm:garm /etc/garm sudo chown -R garm:garm /etc/garm
sudo chmod 750 -R /etc/garm sudo chmod 750 -R /etc/garm
``` ```
Enable the service: Enable the service:
```bash ```bash
sudo systemctl enable garm sudo systemctl enable garm
``` ```
Customize the config in ```/etc/garm/config.toml```, and start the service: Customize the config in ```/etc/garm/config.toml```, and start the service:
```bash ```bash
sudo systemctl start garm sudo systemctl start garm
``` ```
## Configuration ## Configuration
@ -105,9 +106,9 @@ The ```garm``` configuration is a simple ```toml```. A sample of the config file
There are 3 major sections of the config that require your attention: There are 3 major sections of the config that require your attention:
* [Github credentials section](/doc/github_credentials.md) * [Github credentials section](/doc/github_credentials.md)
* [Providers section](/doc/providers.md) * [Providers section](/doc/providers.md)
* [The database section](/doc/database.md) * [The database section](/doc/database.md)
Once you've configured your database, providers and github credentials, you'll need to configure your [webhooks and the callback_url](/doc/webhooks_and_callbacks.md). Once you've configured your database, providers and github credentials, you'll need to configure your [webhooks and the callback_url](/doc/webhooks_and_callbacks.md).
@ -115,12 +116,11 @@ At this point, you should be done. Have a look at the [running garm document](/d
If you would like to use ```garm``` with a different IaaS than the ones already available, have a loot at the [writing an external provider](/doc/external_provider.md) page. If you would like to use ```garm``` with a different IaaS than the ones already available, have a loot at the [writing an external provider](/doc/external_provider.md) page.
## Security considerations ## Security considerations
Garm does not apply any ACLs of any kind to the instances it creates. That task remains in the responsability of the user. [Here is a guide for creating ACLs in LXD](https://linuxcontainers.org/lxd/docs/master/howto/network_acls/). You can of course use ```iptables``` or ```nftables``` to create any rules you wish. I recommend you create a separate isolated lxd bridge for runners, and secure it using ACLs/iptables/nftables. Garm does not apply any ACLs of any kind to the instances it creates. That task remains in the responsibility of the user. [Here is a guide for creating ACLs in LXD](https://linuxcontainers.org/lxd/docs/master/howto/network_acls/). You can of course use ```iptables``` or ```nftables``` to create any rules you wish. I recommend you create a separate isolated lxd bridge for runners, and secure it using ACLs/iptables/nftables.
You must make sure that the code that runs as part of the workflows is trusted, and if that cannot be done, you must make sure that any malitious code that will be pulled in by the actions and run as part of a workload, is as contained as possible. There is a nice article about [securing your workflow runs here](https://blog.gitguardian.com/github-actions-security-cheat-sheet/). You must make sure that the code that runs as part of the workflows is trusted, and if that cannot be done, you must make sure that any malicious code that will be pulled in by the actions and run as part of a workload, is as contained as possible. There is a nice article about [securing your workflow runs here](https://blog.gitguardian.com/github-actions-security-cheat-sheet/).
## Write your own provider ## Write your own provider
@ -128,4 +128,4 @@ The providers are interfaces between ```garm``` and a particular IaaS in which w
There is currently one **native** provider for [LXD](https://linuxcontainers.org/lxd/) and two **external** providers for [Openstack and Azure](/contrib/providers.d/). There is currently one **native** provider for [LXD](https://linuxcontainers.org/lxd/) and two **external** providers for [Openstack and Azure](/contrib/providers.d/).
If you want to write your own provider, you can choose to write a native one, or implement an **external** one. The easiest one to write is probably an **external** provider. Please see the [Writing an external provider](/doc/external_provider.md) document for details. Also, feel free to inspect the two available external providers in this repository. If you want to write your own provider, you can choose to write a native one, or implement an **external** one. The easiest one to write is probably an **external** provider. Please see the [Writing an external provider](/doc/external_provider.md) document for details. Also, feel free to inspect the two available external providers in this repository.

60
doc/database.md
View file

@ -2,36 +2,36 @@
Garm currently supports two database backends: Garm currently supports two database backends:
* SQLite3 * SQLite3
* MySQL * MySQL
You can choose either one of these. For most cases, ```SQLite3``` should do, but feel free to go with MySQL if you wish. You can choose either one of these. For most cases, ```SQLite3``` should do, but feel free to go with MySQL if you wish.
```toml ```toml
[database] [database]
# Turn on/off debugging for database queries. # Turn on/off debugging for database queries.
debug = false debug = false
# Database backend to use. Currently supported backends are: # Database backend to use. Currently supported backends are:
# * sqlite3 # * sqlite3
# * mysql # * mysql
backend = "sqlite3" backend = "sqlite3"
# the passphrase option is a temporary measure by which we encrypt the webhook # the passphrase option is a temporary measure by which we encrypt the webhook
# secret that gets saved to the database, using AES256. In the future, secrets # secret that gets saved to the database, using AES256. In the future, secrets
# will be saved to something like Barbican or Vault, eliminating the need for # will be saved to something like Barbican or Vault, eliminating the need for
# this. # this.
passphrase = "n<$n&P#L*TWqOh95_bN5J1r4mhxY7R84HZ%pvM#1vxJ<7~q%YVsCwU@Z60;7~Djo" passphrase = "n<$n&P#L*TWqOh95_bN5J1r4mhxY7R84HZ%pvM#1vxJ<7~q%YVsCwU@Z60;7~Djo"
[database.mysql] [database.mysql]
# If MySQL is used, these are the credentials and connection information used # If MySQL is used, these are the credentials and connection information used
# to connect to the server instance. # to connect to the server instance.
# database username # database username
username = "" username = ""
# Database password # Database password
password = "" password = ""
# hostname to connect to # hostname to connect to
hostname = "" hostname = ""
# database name # database name
database = "" database = ""
[database.sqlite3] [database.sqlite3]
# Path on disk to the sqlite3 database file. # Path on disk to the sqlite3 database file.
db_file = "/home/runner/file.db" db_file = "/home/runner/file.db"
``` ```

301
doc/external_provider.md
View file

@ -1,19 +1,19 @@
# Writing an external provider # Writing an external provider
External provider enables you to write a fully functional provider, using any scripring or programming language. Garm will then call your executable to manage the lifecycle of the instances hosting the runners. This document describes the API that an executable needs to implement to be usable by ```garm```. External provider enables you to write a fully functional provider, using any scripting or programming language. Garm will then call your executable to manage the lifecycle of the instances hosting the runners. This document describes the API that an executable needs to implement to be usable by ```garm```.
## Environment variables ## Environment variables
When ```garm``` calls your executable, a number of environment variables are set, depending on the operation. There are two environment variable will always be set regardless of operation. Those variables are: When ```garm``` calls your executable, a number of environment variables are set, depending on the operation. There are two environment variable will always be set regardless of operation. Those variables are:
* ```GARM_COMMAND``` * ```GARM_COMMAND```
* ```GARM_PROVIDER_CONFIG_FILE``` * ```GARM_PROVIDER_CONFIG_FILE```
The following are variables that are specific to some operations: The following are variables that are specific to some operations:
* ```GARM_CONTROLLER_ID``` * ```GARM_CONTROLLER_ID```
* ```GARM_POOL_ID``` * ```GARM_POOL_ID```
* ```GARM_INSTANCE_ID``` * ```GARM_INSTANCE_ID```
### The GARM_COMMAND variable ### The GARM_COMMAND variable
@ -54,21 +54,21 @@ esac
The ```GARM_CONTROLLER_ID``` variable is set for two operations: The ```GARM_CONTROLLER_ID``` variable is set for two operations:
* CreateInstance * CreateInstance
* RemoveAllInstances * RemoveAllInstances
This variable contains the ```UUID4``` identifying a ```garm``` installation. Whenever you start up ```garm``` for the first time, a new ```UUID4``` is generated and saved in ```garm's``` database. This ID is meant to be used to track all resources created by ```garm``` within a provider. That way, if you decide to tare it all down, you have a way of identifying what was created by one particular installation of ```garm```. This variable contains the ```UUID4``` identifying a ```garm``` installation. Whenever you start up ```garm``` for the first time, a new ```UUID4``` is generated and saved in ```garm's``` database. This ID is meant to be used to track all resources created by ```garm``` within a provider. That way, if you decide to tare it all down, you have a way of identifying what was created by one particular installation of ```garm```.
This is useful if various teams from your company use the same credentials to access a cloud. You won't accidentally clobber someone else's resources. This is useful if various teams from your company use the same credentials to access a cloud. You won't accidentally clobber someone else's resources.
In most clouds you can attach ```tags``` to resources. You can use the controller ID as one of the tagg suring the ```CreateInstance``` operation. In most clouds you can attach ```tags``` to resources. You can use the controller ID as one of the tags during the ```CreateInstance``` operation.
# The GARM_POOL_ID variable ### The GARM_POOL_ID variable
The ```GARM_POOL_ID``` anvironment variable is an ```UUID4``` describing the pool in which a runner is created. This variable is set in two operations: The ```GARM_POOL_ID``` environment variable is an ```UUID4``` describing the pool in which a runner is created. This variable is set in two operations:
* CreateInstance * CreateInstance
* ListInstances * ListInstances
As is with ```GARM_CONTROLLER_ID```, this ID can also be attached as a tag in most clouds. As is with ```GARM_CONTROLLER_ID```, this ID can also be attached as a tag in most clouds.
@ -76,26 +76,26 @@ As is with ```GARM_CONTROLLER_ID```, this ID can also be attached as a tag in mo
The ```GARM_INSTANCE_ID``` environment variable is used in four operations: The ```GARM_INSTANCE_ID``` environment variable is used in four operations:
* GetInstance * GetInstance
* DeleteInstance * DeleteInstance
* Start * Start
* Stop * Stop
It contains the ```provider_id``` of the instance. The ```provider_id``` is a unique identifier, specific to the IaaS in which the compute resource was created. In OpenStack, it's an ```UUID4```, while in LXD, it's the virtual machine's name. It contains the ```provider_id``` of the instance. The ```provider_id``` is a unique identifier, specific to the IaaS in which the compute resource was created. In OpenStack, it's an ```UUID4```, while in LXD, it's the virtual machine's name.
We need this ID whenever we need to execute an operation that targets one specific runner. We need this ID whenever we need to execute an operation that targets one specific runner.
# Operations ## Operations
The operations that a provider must implement are described in the ```Provider``` interface available [here](https://github.com/cloudbase/garm/blob/main/runner/common/provider.go#L22-L39). The external provider implements this interface, and delegates each operation to your external executable. These operations are: The operations that a provider must implement are described in the ```Provider``` interface available [here](https://github.com/cloudbase/garm/blob/main/runner/common/provider.go#L22-L39). The external provider implements this interface, and delegates each operation to your external executable. These operations are:
* CreateInstance * CreateInstance
* DeleteInstance * DeleteInstance
* GetInstance * GetInstance
* ListInstances * ListInstances
* RemoveAllInstances * RemoveAllInstances
* Stop * Stop
* Start * Start
The ```AsParams()``` function does not need to be implemented by the external executable. The ```AsParams()``` function does not need to be implemented by the external executable.
@ -115,101 +115,101 @@ Your external provider will need to be able to handle both. The instance name ge
The ```CreateInstance``` command is the only command that receives information using, environment variables and standard input. The available environment variables are: The ```CreateInstance``` command is the only command that receives information using, environment variables and standard input. The available environment variables are:
* GARM_PROVIDER_CONFIG_FILE - Config file specific to your executable * GARM_PROVIDER_CONFIG_FILE - Config file specific to your executable
* GARM_COMMAND - the command we need to run * GARM_COMMAND - the command we need to run
* GARM_CONTROLLER_ID - The unique ID of the ```garm``` installation * GARM_CONTROLLER_ID - The unique ID of the ```garm``` installation
* GARM_POOL_ID - The unique ID of the pool this node is a part of * GARM_POOL_ID - The unique ID of the pool this node is a part of
The information sent in via standard input is a ```json``` serialized instance of the [BootstrapInstance structure](https://github.com/cloudbase/garm/blob/main/params/params.go#L80-L103) The information sent in via standard input is a ```json``` serialized instance of the [BootstrapInstance structure](https://github.com/cloudbase/garm/blob/main/params/params.go#L80-L103)
Here is a sample of that: Here is a sample of that:
```json ```json
{ {
"name": "garm-e73542f6-2c10-48bb-bfe7-a0374618f405", "name": "garm-e73542f6-2c10-48bb-bfe7-a0374618f405",
"tools": [ "tools": [
{ {
"os": "osx", "os": "osx",
"architecture": "x64", "architecture": "x64",
"download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-osx-x64-2.299.1.tar.gz", "download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-osx-x64-2.299.1.tar.gz",
"filename": "actions-runner-osx-x64-2.299.1.tar.gz", "filename": "actions-runner-osx-x64-2.299.1.tar.gz",
"sha256_checksum": "b0128120f2bc48e5f24df513d77d1457ae845a692f60acf3feba63b8d01a8fdc" "sha256_checksum": "b0128120f2bc48e5f24df513d77d1457ae845a692f60acf3feba63b8d01a8fdc"
}, },
{ {
"os": "linux", "os": "linux",
"architecture": "x64", "architecture": "x64",
"download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-linux-x64-2.299.1.tar.gz", "download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-linux-x64-2.299.1.tar.gz",
"filename": "actions-runner-linux-x64-2.299.1.tar.gz", "filename": "actions-runner-linux-x64-2.299.1.tar.gz",
"sha256_checksum": "147c14700c6cb997421b9a239c012197f11ea9854cd901ee88ead6fe73a72c74" "sha256_checksum": "147c14700c6cb997421b9a239c012197f11ea9854cd901ee88ead6fe73a72c74"
}, },
{ {
"os": "win", "os": "win",
"architecture": "x64", "architecture": "x64",
"download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-win-x64-2.299.1.zip", "download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-win-x64-2.299.1.zip",
"filename": "actions-runner-win-x64-2.299.1.zip", "filename": "actions-runner-win-x64-2.299.1.zip",
"sha256_checksum": "f7940b16451d6352c38066005f3ee6688b53971fcc20e4726c7907b32bfdf539" "sha256_checksum": "f7940b16451d6352c38066005f3ee6688b53971fcc20e4726c7907b32bfdf539"
}, },
{ {
"os": "linux", "os": "linux",
"architecture": "arm", "architecture": "arm",
"download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-linux-arm-2.299.1.tar.gz", "download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-linux-arm-2.299.1.tar.gz",
"filename": "actions-runner-linux-arm-2.299.1.tar.gz", "filename": "actions-runner-linux-arm-2.299.1.tar.gz",
"sha256_checksum": "a4d66a766ff3b9e07e3e068a1d88b04e51c27c9b94ae961717e0a5f9ada998e6" "sha256_checksum": "a4d66a766ff3b9e07e3e068a1d88b04e51c27c9b94ae961717e0a5f9ada998e6"
}, },
{ {
"os": "linux", "os": "linux",
"architecture": "arm64", "architecture": "arm64",
"download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-linux-arm64-2.299.1.tar.gz", "download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-linux-arm64-2.299.1.tar.gz",
"filename": "actions-runner-linux-arm64-2.299.1.tar.gz", "filename": "actions-runner-linux-arm64-2.299.1.tar.gz",
"sha256_checksum": "debe1cc9656963000a4fbdbb004f475ace5b84360ace2f7a191c1ccca6a16c00" "sha256_checksum": "debe1cc9656963000a4fbdbb004f475ace5b84360ace2f7a191c1ccca6a16c00"
}, },
{ {
"os": "osx", "os": "osx",
"architecture": "arm64", "architecture": "arm64",
"download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-osx-arm64-2.299.1.tar.gz", "download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-osx-arm64-2.299.1.tar.gz",
"filename": "actions-runner-osx-arm64-2.299.1.tar.gz", "filename": "actions-runner-osx-arm64-2.299.1.tar.gz",
"sha256_checksum": "f73849b9a78459d2e08b9d3d2f60464a55920de120e228b0645b01abe68d9072" "sha256_checksum": "f73849b9a78459d2e08b9d3d2f60464a55920de120e228b0645b01abe68d9072"
}, },
{ {
"os": "win", "os": "win",
"architecture": "arm64", "architecture": "arm64",
"download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-win-arm64-2.299.1.zip", "download_url": "https://github.com/actions/runner/releases/download/v2.299.1/actions-runner-win-arm64-2.299.1.zip",
"filename": "actions-runner-win-arm64-2.299.1.zip", "filename": "actions-runner-win-arm64-2.299.1.zip",
"sha256_checksum": "d1a9d8209f03589c8dc05ee17ae8d194756377773a4010683348cdd6eefa2da7" "sha256_checksum": "d1a9d8209f03589c8dc05ee17ae8d194756377773a4010683348cdd6eefa2da7"
} }
], ],
"repo_url": "https://github.com/gabriel-samfira/scripts", "repo_url": "https://github.com/gabriel-samfira/scripts",
"callback-url": "https://garm.example.com/api/v1/callbacks/status", "callback-url": "https://garm.example.com/api/v1/callbacks/status",
"metadata-url": "https://garm.example.com/api/v1/metadata", "metadata-url": "https://garm.example.com/api/v1/metadata",
"instance-token": "super secret JWT token", "instance-token": "super secret JWT token",
"ssh-keys": null, "ssh-keys": null,
"ca-cert-bundle": null, "ca-cert-bundle": null,
"arch": "amd64", "arch": "amd64",
"flavor": "m1.small", "flavor": "m1.small",
"image": "8ed8a690-69b6-49eb-982f-dcb466895e2d", "image": "8ed8a690-69b6-49eb-982f-dcb466895e2d",
"labels": [ "labels": [
"ubuntu", "ubuntu",
"self-hosted", "self-hosted",
"x64", "x64",
"linux", "linux",
"openstack", "openstack",
"runner-controller-id:f9286791-1589-4f39-a106-5b68c2a18af4", "runner-controller-id:f9286791-1589-4f39-a106-5b68c2a18af4",
"runner-pool-id:9dcf590a-1192-4a9c-b3e4-e0902974c2c0" "runner-pool-id:9dcf590a-1192-4a9c-b3e4-e0902974c2c0"
], ],
"pool_id": "9dcf590a-1192-4a9c-b3e4-e0902974c2c0" "pool_id": "9dcf590a-1192-4a9c-b3e4-e0902974c2c0"
} }
``` ```
In your executable you can read in this blob, by using something like this: In your executable you can read in this blob, by using something like this:
```bash ```bash
# Test if the stdin file descriptor is opened # Test if the stdin file descriptor is opened
if [ ! -t 0 ] if [ ! -t 0 ]
then then
# Read in the information from standard in # Read in the information from standard in
INPUT=$(cat -) INPUT=$(cat -)
fi fi
``` ```
Then you can easily parse it. If you're using ```bash```, you can use the amazing [jq json processor](https://stedolan.github.io/jq/). Other programming languages have suitable libraries that can handle ```json```. Then you can easily parse it. If you're using ```bash```, you can use the amazing [jq json processor](https://stedolan.github.io/jq/). Other programming languages have suitable libraries that can handle ```json```.
@ -219,29 +219,29 @@ Refer to the OpenStack or Azure providers available in the [providers.d](../cont
### CreateInstance outputs ### CreateInstance outputs
On success, your executable is expected to print to standard output a json that can be unserialized into an ```Instance{}``` structure [defined here](https://github.com/cloudbase/garm/blob/main/params/params.go#L43-L78). On success, your executable is expected to print to standard output a json that can be deserialized into an ```Instance{}``` structure [defined here](https://github.com/cloudbase/garm/blob/main/params/params.go#L43-L78).
Not all fields are expected to be populated by the provider. The ones that should be set are: Not all fields are expected to be populated by the provider. The ones that should be set are:
```json ```json
{ {
"provider_id": "88818ff3-1fca-4cb5-9b37-84bfc3511ea6", "provider_id": "88818ff3-1fca-4cb5-9b37-84bfc3511ea6",
"name": "garm-0542a982-4a0d-4aca-aef0-d736c96f61ca", "name": "garm-0542a982-4a0d-4aca-aef0-d736c96f61ca",
"os_type": "linux", "os_type": "linux",
"os_name": "ubuntu", "os_name": "ubuntu",
"os_version": "20.04", "os_version": "20.04",
"os_arch": "x86_64", "os_arch": "x86_64",
"status": "running", "status": "running",
"pool_id": "41c4a43a-acee-493a-965b-cf340b2c775d", "pool_id": "41c4a43a-acee-493a-965b-cf340b2c775d",
"provider_fault": "" "provider_fault": ""
} }
``` ```
In case of error, ```garm``` expects at the very least to see a non-zero exit code. If possible, your executable should return as much information as possible via the above ```json```, with the ```status``` field set to ```error``` and the ```provider_fault``` set to a meaningful error message describing what has happened. That information will be visible when doing a: In case of error, ```garm``` expects at the very least to see a non-zero exit code. If possible, your executable should return as much information as possible via the above ```json```, with the ```status``` field set to ```error``` and the ```provider_fault``` set to a meaningful error message describing what has happened. That information will be visible when doing a:
```bash ```bash
garm-cli runner show <runner name> garm-cli runner show <runner name>
``` ```
## DeleteInstance ## DeleteInstance
@ -249,9 +249,9 @@ The ```DeleteInstance``` command will permanently remove an instance from the cl
Available environment variables: Available environment variables:
* GARM_COMMAND * GARM_COMMAND
* GARM_PROVIDER_CONFIG_FILE * GARM_PROVIDER_CONFIG_FILE
* GARM_INSTANCE_ID * GARM_INSTANCE_ID
This command is not expected to output anything. On success it should simply ```exit 0```. This command is not expected to output anything. On success it should simply ```exit 0```.
@ -263,23 +263,23 @@ The ```GetInstance``` command will return details about the instance, as seen by
Available environment variables: Available environment variables:
* GARM_COMMAND * GARM_COMMAND
* GARM_PROVIDER_CONFIG_FILE * GARM_PROVIDER_CONFIG_FILE
* GARM_INSTANCE_ID * GARM_INSTANCE_ID
On success, this command is expected to return a valid ```json``` that can be unserialized into an ```Instance{}``` structure (see CreateInstance). If possible, IP addresses allocated to the VM should be returned in adition to the sample ```json``` printed above. On success, this command is expected to return a valid ```json``` that can be deserialized into an ```Instance{}``` structure (see CreateInstance). If possible, IP addresses allocated to the VM should be returned in addition to the sample ```json``` printed above.
On failure, this command is expected to return a non-zero exit code. On failure, this command is expected to return a non-zero exit code.
## ListInstances ## ListInstances
The ```ListInstances``` command will print to standard output, a json that is unserializable into an **array** of ```Instance{}```. The ```ListInstances``` command will print to standard output, a json that is deserializable into an **array** of ```Instance{}```.
Available environment variables: Available environment variables:
* GARM_COMMAND * GARM_COMMAND
* GARM_PROVIDER_CONFIG_FILE * GARM_PROVIDER_CONFIG_FILE
* GARM_POOL_ID * GARM_POOL_ID
This command must list all instances that have been tagged with the value in ```GARM_POOL_ID```. This command must list all instances that have been tagged with the value in ```GARM_POOL_ID```.
@ -293,9 +293,9 @@ The ```RemoveAllInstances``` operation will remove all resources created in a cl
Available environment variables: Available environment variables:
* GARM_COMMAND * GARM_COMMAND
* GARM_PROVIDER_CONFIG_FILE * GARM_PROVIDER_CONFIG_FILE
* GARM_CONTROLLER_ID * GARM_CONTROLLER_ID
On success, no output is expected. On success, no output is expected.
@ -307,15 +307,14 @@ The ```Start``` operation will start the virtual machine in the selected cloud.
Available environment variables: Available environment variables:
* GARM_COMMAND * GARM_COMMAND
* GARM_PROVIDER_CONFIG_FILE * GARM_PROVIDER_CONFIG_FILE
* GARM_INSTANCE_ID * GARM_INSTANCE_ID
On success, no output is expected. On success, no output is expected.
On failure, a non-zero exit code is expected. On failure, a non-zero exit code is expected.
## Stop ## Stop
NOTE: This operation is currently not use by ```garm```, but should be implemented. NOTE: This operation is currently not use by ```garm```, but should be implemented.
@ -324,9 +323,9 @@ The ```Stop``` operation will stop the virtual machine in the selected cloud.
Available environment variables: Available environment variables:
* GARM_COMMAND * GARM_COMMAND
* GARM_PROVIDER_CONFIG_FILE * GARM_PROVIDER_CONFIG_FILE
* GARM_INSTANCE_ID * GARM_INSTANCE_ID
On success, no output is expected. On success, no output is expected.

14
doc/github_credentials.md
View file

@ -4,16 +4,14 @@ The ```github``` config section holds credentials and API endpoint information f
Tying the API endpoint info to the credentials allows us to use the same ```garm``` installation with both [github.com](https://github.com) and private deployments. All you have to do is to add the needed endpoint info (see bellow). Tying the API endpoint info to the credentials allows us to use the same ```garm``` installation with both [github.com](https://github.com) and private deployments. All you have to do is to add the needed endpoint info (see bellow).
Garm uses a [Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) to create runner registration tokens, list current self hosted runners and potentially remove them if they become orphaned (the VM was manually removed on the provider). Garm uses a [Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) to create runner registration tokens, list current self hosted runners and potentially remove them if they become orphaned (the VM was manually removed on the provider).
From the list of scopes, you will need to select: From the list of scopes, you will need to select:
* ```public_repo``` - for access to a repository * ```public_repo``` - for access to a repository
* ```repo``` - for access to a private repository * ```repo``` - for access to a private repository
* ```admin:org``` - if you plan on using this with an organization to which you have access * ```admin:org``` - if you plan on using this with an organization to which you have access
* ```manage_runners:enterprise``` - if you plan to use garm at the enterprise level * ```manage_runners:enterprise``` - if you plan to use garm at the enterprise level
The resulting token must be configured in the ```[[github]]``` section of the config. Sample as follows: The resulting token must be configured in the ```[[github]]``` section of the config. Sample as follows:
@ -44,10 +42,10 @@ The resulting token must be configured in the ```[[github]]``` section of the co
# ca_cert_bundle (optional) is the CA certificate bundle in PEM format that will be used by the github # ca_cert_bundle (optional) is the CA certificate bundle in PEM format that will be used by the github
# client to talk to the API. This bundle will also be sent to all runners as bootstrap params. # client to talk to the API. This bundle will also be sent to all runners as bootstrap params.
# Use this option if you're using a self signed certificate. # Use this option if you're using a self signed certificate.
# Leave this blank if you're using github.com or if your certificare is signed by a valid CA. # Leave this blank if you're using github.com or if your certificate is signed by a valid CA.
ca_cert_bundle = "/etc/garm/ghe.crt" ca_cert_bundle = "/etc/garm/ghe.crt"
``` ```
The double paranthesis means that this is an array. You can specify the ```[[github]]``` section multiple times, with different tokens from different users, or with different access levels. You will then be able to list the available credentials using the API, and reference these credentials when adding repositories or organizations. The double parenthesis means that this is an array. You can specify the ```[[github]]``` section multiple times, with different tokens from different users, or with different access levels. You will then be able to list the available credentials using the API, and reference these credentials when adding repositories or organizations.
The API will only ever return the name and description to the API consumer. The API will only ever return the name and description to the API consumer.

29
doc/providers.md
View file

@ -1,8 +1,9 @@
# Provider configuration # Provider configuration
Garm was designed to be extensible. The database layer as well as the providers are defined as interfaces. Currently there are two providers: Garm was designed to be extensible. The database layer as well as the providers are defined as interfaces. Currently there are two providers:
* [LXD](https://linuxcontainers.org/lxd/introduction/)
* External * [LXD](https://linuxcontainers.org/lxd/introduction/)
* External
LXD is the simplest cloud-like system you can easily set up on any GNU/Linux machine, which enables you to create both containers and Virtual Machines. The ```external``` provider is a special type of provider, which delegates functionality to external executables. LXD is the simplest cloud-like system you can easily set up on any GNU/Linux machine, which enables you to create both containers and Virtual Machines. The ```external``` provider is a special type of provider, which delegates functionality to external executables.
@ -22,7 +23,7 @@ Garm leverages the virtual machines feature of LXD to create the runners. Here i
name = "lxd_local" name = "lxd_local"
# Provider type. Garm is designed to allow creating providers which are used to spin # Provider type. Garm is designed to allow creating providers which are used to spin
# up compute resources, which in turn will run the github runner software. # up compute resources, which in turn will run the github runner software.
# Currently, LXD is the only supprted provider, but more will be written in the future. # Currently, LXD is the only supported provider, but more will be written in the future.
provider_type = "lxd" provider_type = "lxd"
# A short description of this provider. The name, description and provider types will # A short description of this provider. The name, description and provider types will
# be included in the information returned by the API when listing available providers. # be included in the information returned by the API when listing available providers.
@ -91,22 +92,21 @@ By default, garm does not load any image remotes. You get to choose which remote
The sample config file in this repository has the usual default ```LXD``` remotes: The sample config file in this repository has the usual default ```LXD``` remotes:
* https://cloud-images.ubuntu.com/releases (ubuntu) - Official Ubuntu images * <https://cloud-images.ubuntu.com/releases> (ubuntu) - Official Ubuntu images
* https://cloud-images.ubuntu.com/daily (ubuntu_daily) - Official Ubuntu images, daily build * <https://cloud-images.ubuntu.com/daily> (ubuntu_daily) - Official Ubuntu images, daily build
* https://images.linuxcontainers.org (images) - Comunity maintained images for various operating systems * <https://images.linuxcontainers.org> (images) - Community maintained images for various operating systems
When creating a new pool, you'll be able to specify which image you want to use. The images are referenced by ```remote_name:image_tag```. For example, if you want to launch a runner on an Ubuntu 20.04, the image name would be ```ubuntu:20.04```. For a daily image it would be ```ubuntu_daily:20.04```. And for one of the unnoficial images it would be ```images:centos/8-Stream/cloud```. Note, for unofficial images you need to use the tags that have ```/cloud``` in the name. These images come pre-installed with ```cloud-init``` which we need to set up the runners automatically. When creating a new pool, you'll be able to specify which image you want to use. The images are referenced by ```remote_name:image_tag```. For example, if you want to launch a runner on an Ubuntu 20.04, the image name would be ```ubuntu:20.04```. For a daily image it would be ```ubuntu_daily:20.04```. And for one of the unofficial images it would be ```images:centos/8-Stream/cloud```. Note, for unofficial images you need to use the tags that have ```/cloud``` in the name. These images come pre-installed with ```cloud-init``` which we need to set up the runners automatically.
You can also create your own image remote, where you can host your own custom images. If you want to build your own images, have a look at [distrobuilder](https://github.com/lxc/distrobuilder). You can also create your own image remote, where you can host your own custom images. If you want to build your own images, have a look at [distrobuilder](https://github.com/lxc/distrobuilder).
Image remotes in the ```garm``` config, is a map of strings to remote settins. The name of the remote is the last bit of string in the section header. For example, the following section ```[provider.lxd.image_remotes.ubuntu_daily]```, defines the image remote named **ubuntu_daily**. Use this name to reference images inside that remote. Image remotes in the ```garm``` config, is a map of strings to remote settings. The name of the remote is the last bit of string in the section header. For example, the following section ```[provider.lxd.image_remotes.ubuntu_daily]```, defines the image remote named **ubuntu_daily**. Use this name to reference images inside that remote.
## The External provider ## The External provider
The external provider is a special kind of provider. It delegates the functionality needed to create the runners to external executables. These executables can be either binaries or scripts. As long as they adhere to the needed interface, they can be used to create runners in any target IaaS. This is identical to what ```containerd``` does with ```CNIs```. The external provider is a special kind of provider. It delegates the functionality needed to create the runners to external executables. These executables can be either binaries or scripts. As long as they adhere to the needed interface, they can be used to create runners in any target IaaS. This is identical to what ```containerd``` does with ```CNIs```.
There is currently one external provider for [OpenStack](https://www.openstack.org/) available in the [contrb folder of this repository](../contrib/providers.d/openstack). The provider is written in ```bash``` and it is just a sample. A production ready provider would need more error checking and idempotency, but it serves as an example of what can be done. As it stands, it is functional. There is currently one external provider for [OpenStack](https://www.openstack.org/) available in the [contrib folder of this repository](../contrib/providers.d/openstack). The provider is written in ```bash``` and it is just a sample. A production ready provider would need more error checking and idempotency, but it serves as an example of what can be done. As it stands, it is functional.
The configuration for an external provider is quite simple: The configuration for an external provider is quite simple:
@ -129,12 +129,11 @@ provider_type = "external"
The external provider has three options: The external provider has three options:
* ```provider_executable``` * ```provider_executable```
* ```config_file``` * ```config_file```
The ```provider_executable``` option is the absolute path to an executable that implements the provider logic. Garm will delegate all provider operations to this executable. This executable can be anything (bash, python, perl, go, etc). See [Writing an external provider](./external_provider.md) for more details. The ```provider_executable``` option is the absolute path to an executable that implements the provider logic. Garm will delegate all provider operations to this executable. This executable can be anything (bash, python, perl, go, etc). See [Writing an external provider](./external_provider.md) for more details.
The ```config_file``` option is a path on disk to an arbitrary file, that is passed to the external executable via the environment variable ```GARM_PROVIDER_CONFIG_FILE```. This file is only relevant to the external provider. Garm itself does not read it. In the case of the OpenStack provider, this file contains access information for an OpenStack cloud (what you would tipically find in a ```keystonerc``` file) as well as some provider specific options like whether or not to boot from volume and which tenant network to use. You can check out the [sample config file](../contrib/providers.d/openstack/keystonerc) in this repository. The ```config_file``` option is a path on disk to an arbitrary file, that is passed to the external executable via the environment variable ```GARM_PROVIDER_CONFIG_FILE```. This file is only relevant to the external provider. Garm itself does not read it. In the case of the OpenStack provider, this file contains access information for an OpenStack cloud (what you would typically find in a ```keystonerc``` file) as well as some provider specific options like whether or not to boot from volume and which tenant network to use. You can check out the [sample config file](../contrib/providers.d/openstack/keystonerc) in this repository.
If you want to implement an external provider, you can use this file for anything you need to pass into the binary when ```garm``` calls it to execute a particular operation. If you want to implement an external provider, you can use this file for anything you need to pass into the binary when ```garm``` calls it to execute a particular operation.

582
doc/running_garm.md
View file

@ -1,22 +1,22 @@
## Running garm # Running garm
Create a folder for the config: Create a folder for the config:
```bash ```bash
mkdir $HOME/garm mkdir $HOME/garm
``` ```
Create a config file for ```garm```: Create a config file for ```garm```:
```bash ```bash
cp ./testdata/config.toml $HOME/garm/config.toml cp ./testdata/config.toml $HOME/garm/config.toml
``` ```
Customize the config whichever way you want, then run ```garm```: Customize the config whichever way you want, then run ```garm```:
```bash ```bash
garm -config $HOME/garm/config.toml garm -config $HOME/garm/config.toml
``` ```
This will start the API and migrate the database. Note, if you're using MySQL, you will need to create a database, grant access to a user and configure those credentials in the ```config.toml``` file. This will start the API and migrate the database. Note, if you're using MySQL, you will need to create a database, grant access to a user and configure those credentials in the ```config.toml``` file.
@ -24,20 +24,20 @@ This will start the API and migrate the database. Note, if you're using MySQL, y
Before you can use ```garm```, you need to initialize it. This means we need to create an admin user, and login: Before you can use ```garm```, you need to initialize it. This means we need to create an admin user, and login:
```bash ```bash
ubuntu@experiments:~$ garm-cli init --name="local_garm" --url https://garm.example.com ubuntu@experiments:~$ garm-cli init --name="local_garm" --url https://garm.example.com
Username: admin Username: admin
Email: root@localhost Email: root@localhost
✔ Password: *************█ ✔ Password: *************█
+----------+--------------------------------------+ +----------+--------------------------------------+
| FIELD | VALUE | | FIELD | VALUE |
+----------+--------------------------------------+ +----------+--------------------------------------+
| ID | ef4ab6fd-1252-4d5a-ba5a-8e8bd01610ae | | ID | ef4ab6fd-1252-4d5a-ba5a-8e8bd01610ae |
| Username | admin | | Username | admin |
| Email | root@localhost | | Email | root@localhost |
| Enabled | true | | Enabled | true |
+----------+--------------------------------------+ +----------+--------------------------------------+
``` ```
Alternatively you can run this in non-interactive mode. See ```garm-cli init -h``` for details. Alternatively you can run this in non-interactive mode. See ```garm-cli init -h``` for details.
@ -45,104 +45,104 @@ Alternatively you can run this in non-interactive mode. See ```garm-cli init -h`
Before we begin, let's make our lives a little easier and set up bash completion. The wonderful [cobra](https://github.com/spf13/cobra) library gives us completion for free: Before we begin, let's make our lives a little easier and set up bash completion. The wonderful [cobra](https://github.com/spf13/cobra) library gives us completion for free:
```bash ```bash
mkdir $HOME/.bash_completion.d mkdir $HOME/.bash_completion.d
echo 'source $HOME/.bash_completion.d/* >/dev/null 2>&1|| true' >> $HOME/.bash_completion echo 'source $HOME/.bash_completion.d/* >/dev/null 2>&1|| true' >> $HOME/.bash_completion
``` ```
Now generate the completion file: Now generate the completion file:
```bash ```bash
garm-cli completion bash > $HOME/.bash_completion.d/garm garm-cli completion bash > $HOME/.bash_completion.d/garm
``` ```
Completion for multipiple shells is available: Completion for multiple shells is available:
```bash ```bash
ubuntu@experiments:~$ garm-cli completion ubuntu@experiments:~$ garm-cli completion
Generate the autocompletion script for garm-cli for the specified shell. Generate the autocompletion script for garm-cli for the specified shell.
See each sub-command's help for details on how to use the generated script. See each sub-command's help for details on how to use the generated script.
Usage: Usage:
garm-cli completion [command] garm-cli completion [command]
Available Commands: Available Commands:
bash Generate the autocompletion script for bash bash Generate the autocompletion script for bash
fish Generate the autocompletion script for fish fish Generate the autocompletion script for fish
powershell Generate the autocompletion script for powershell powershell Generate the autocompletion script for powershell
zsh Generate the autocompletion script for zsh zsh Generate the autocompletion script for zsh
Flags: Flags:
-h, --help help for completion -h, --help help for completion
Global Flags: Global Flags:
--debug Enable debug on all API calls --debug Enable debug on all API calls
Use "garm-cli completion [command] --help" for more information about a command. Use "garm-cli completion [command] --help" for more information about a command.
``` ```
## Adding a repository/organization/enterprise ## Adding a repository/organization/enterprise
To add a repository, we need credentials. Let's list the available credentials currently configured. These credentials are added to ```garm``` using the config file (see above), but we need to reference them by name when creating a repo. To add a repository, we need credentials. Let's list the available credentials currently configured. These credentials are added to ```garm``` using the config file (see above), but we need to reference them by name when creating a repo.
```bash ```bash
ubuntu@experiments:~$ garm-cli credentials list ubuntu@experiments:~$ garm-cli credentials list
+---------+------------------------------+ +---------+------------------------------+
| NAME | DESCRIPTION | | NAME | DESCRIPTION |
+---------+------------------------------+ +---------+------------------------------+
| gabriel | github token or user gabriel | | gabriel | github token or user gabriel |
+---------+------------------------------+ +---------+------------------------------+
``` ```
Now we can add a repository to ```garm```: Now we can add a repository to ```garm```:
```bash ```bash
ubuntu@experiments:~$ garm-cli repository create \ ubuntu@experiments:~$ garm-cli repository create \
--credentials=gabriel \ --credentials=gabriel \
--owner=gabriel-samfira \ --owner=gabriel-samfira \
--name=scripts \ --name=scripts \
--webhook-secret="super secret webhook secret you configured in github webhooks" --webhook-secret="super secret webhook secret you configured in github webhooks"
+-------------+--------------------------------------+ +-------------+--------------------------------------+
| FIELD | VALUE | | FIELD | VALUE |
+-------------+--------------------------------------+ +-------------+--------------------------------------+
| ID | 77258e1b-81d2-4821-bdd7-f6923a026455 | | ID | 77258e1b-81d2-4821-bdd7-f6923a026455 |
| Owner | gabriel-samfira | | Owner | gabriel-samfira |
| Name | scripts | | Name | scripts |
| Credentials | gabriel | | Credentials | gabriel |
+-------------+--------------------------------------+ +-------------+--------------------------------------+
``` ```
To add an organization, use the following command: To add an organization, use the following command:
```bash ```bash
ubuntu@experiments:~$ garm-cli organization create \ ubuntu@experiments:~$ garm-cli organization create \
--credentials=gabriel \ --credentials=gabriel \
--name=gsamfira \ --name=gsamfira \
--webhook-secret="$SECRET" --webhook-secret="$SECRET"
+-------------+--------------------------------------+ +-------------+--------------------------------------+
| FIELD | VALUE | | FIELD | VALUE |
+-------------+--------------------------------------+ +-------------+--------------------------------------+
| ID | 7f0b83d5-3dc0-42de-b189-f9bbf1ae8901 | | ID | 7f0b83d5-3dc0-42de-b189-f9bbf1ae8901 |
| Name | gsamfira | | Name | gsamfira |
| Credentials | gabriel | | Credentials | gabriel |
+-------------+--------------------------------------+ +-------------+--------------------------------------+
``` ```
To add an enterprise, use the following command: To add an enterprise, use the following command:
```bash ```bash
ubuntu@experiments:~$ garm-cli enterprise create \ ubuntu@experiments:~$ garm-cli enterprise create \
--credentials=gabriel \ --credentials=gabriel \
--name=gsamfira \ --name=gsamfira \
--webhook-secret="$SECRET" --webhook-secret="$SECRET"
+-------------+--------------------------------------+ +-------------+--------------------------------------+
| FIELD | VALUE | | FIELD | VALUE |
+-------------+--------------------------------------+ +-------------+--------------------------------------+
| ID | 0925033b-049f-4334-a460-c26f979d2356 | | ID | 0925033b-049f-4334-a460-c26f979d2356 |
| Name | gsamfira | | Name | gsamfira |
| Credentials | gabriel | | Credentials | gabriel |
+-------------+--------------------------------------+ +-------------+--------------------------------------+
``` ```
## Creating a pool ## Creating a pool
@ -150,244 +150,242 @@ Pools are objects that define one type of worker and rules by which that pool of
Before we can create a pool, we need to list the available providers. Providers are defined in the config (see above), but we need to reference them by name in the pool. Before we can create a pool, we need to list the available providers. Providers are defined in the config (see above), but we need to reference them by name in the pool.
```bash ```bash
ubuntu@experiments:~$ garm-cli provider list ubuntu@experiments:~$ garm-cli provider list
+-----------+------------------------+------+ +-----------+------------------------+------+
| NAME | DESCRIPTION | TYPE | | NAME | DESCRIPTION | TYPE |
+-----------+------------------------+------+ +-----------+------------------------+------+
| lxd_local | Local LXD installation | lxd | | lxd_local | Local LXD installation | lxd |
+-----------+------------------------+------+ +-----------+------------------------+------+
``` ```
Now we can create a pool for repo ```gabriel-samfira/scripts```: Now we can create a pool for repo ```gabriel-samfira/scripts```:
```bash ```bash
ubuntu@experiments:~$ garm-cli pool add \ ubuntu@experiments:~$ garm-cli pool add \
--repo=77258e1b-81d2-4821-bdd7-f6923a026455 \ --repo=77258e1b-81d2-4821-bdd7-f6923a026455 \
--flavor="default" \ --flavor="default" \
--image="ubuntu:20.04" \ --image="ubuntu:20.04" \
--provider-name="lxd_local" \ --provider-name="lxd_local" \
--tags="ubuntu,simple-runner,repo-runner" \ --tags="ubuntu,simple-runner,repo-runner" \
--enabled=false --enabled=false
+------------------+-------------------------------------------------------------+ +------------------+-------------------------------------------------------------+
| FIELD | VALUE | | FIELD | VALUE |
+------------------+-------------------------------------------------------------+ +------------------+-------------------------------------------------------------+
| ID | fb25f308-7ad2-4769-988e-6ec2935f642a | | ID | fb25f308-7ad2-4769-988e-6ec2935f642a |
| Provider Name | lxd_local | | Provider Name | lxd_local |
| Image | ubuntu:20.04 | | Image | ubuntu:20.04 |
| Flavor | default | | Flavor | default |
| OS Type | linux | | OS Type | linux |
| OS Architecture | amd64 | | OS Architecture | amd64 |
| Max Runners | 5 | | Max Runners | 5 |
| Min Idle Runners | 1 | | Min Idle Runners | 1 |
| Tags | ubuntu, simple-runner, repo-runner, self-hosted, x64, linux | | Tags | ubuntu, simple-runner, repo-runner, self-hosted, x64, linux |
| Belongs to | gabriel-samfira/scripts | | Belongs to | gabriel-samfira/scripts |
| Level | repo | | Level | repo |
| Enabled | false | | Enabled | false |
+------------------+-------------------------------------------------------------+ +------------------+-------------------------------------------------------------+
``` ```
There are a bunch of things going on here, so let's break it down. We created a pool for repo ```gabriel-samfira/scripts``` (identified by the ID ```77258e1b-81d2-4821-bdd7-f6923a026455```). This pool has the following characteristics: There are a bunch of things going on here, so let's break it down. We created a pool for repo ```gabriel-samfira/scripts``` (identified by the ID ```77258e1b-81d2-4821-bdd7-f6923a026455```). This pool has the following characteristics:
* flavor=default - The **flavor** describes the hardware aspects of an instance. In LXD terms, this translates to [profiles](https://linuxcontainers.org/lxd/docs/master/profiles/). In LXD, profiles describe how much memory, CPU, NICs and disks a particular instance will get. Much like the flavors in OpenStack or any public cloud provider * flavor=default - The **flavor** describes the hardware aspects of an instance. In LXD terms, this translates to [profiles](https://linuxcontainers.org/lxd/docs/master/profiles/). In LXD, profiles describe how much memory, CPU, NICs and disks a particular instance will get. Much like the flavors in OpenStack or any public cloud provider
* image=ubuntu:20.04 - The image describes the operating system that will be spun up on the provider. LXD fetches these images from one of the configured remotes, or from the locally cached images. On AWS, this would be an AMI (for example). * image=ubuntu:20.04 - The image describes the operating system that will be spun up on the provider. LXD fetches these images from one of the configured remotes, or from the locally cached images. On AWS, this would be an AMI (for example).
* provider-name=lxd_local - This is the provider on which we'll be spinning up runners. You can have as many providers defined as you wish, and you can reference either one of them when creating a pool. * provider-name=lxd_local - This is the provider on which we'll be spinning up runners. You can have as many providers defined as you wish, and you can reference either one of them when creating a pool.
* tags="ubuntu,simple-runner,repo-runner" - This list of tags will be added to all runners maintained by this pool. These are the tags you can use to target whese runners in your workflows. By default, the github runner will automatically add a few default tags (self-hosted, x64, linux in the above example) * tags="ubuntu,simple-runner,repo-runner" - This list of tags will be added to all runners maintained by this pool. These are the tags you can use to target these runners in your workflows. By default, the github runner will automatically add a few default tags (self-hosted, x64, linux in the above example)
* enabled=false - This option creates the pool in **disabled** state. When disabled, no new runners will be spun up. * enabled=false - This option creates the pool in **disabled** state. When disabled, no new runners will be spun up.
By default, a pool is created with a max worker count of ```5``` and a minimum idle runner count of ```1```. This means that this pool will create by default one runner, and will automatically add more, as jobs are triggered on github. The idea is to have at least one runner ready to accept a workflow job. The pool will keep adding workers until the max runner count is reached. Once a workflow job is complete, the runner is automatically deleted, and replaced. By default, a pool is created with a max worker count of ```5``` and a minimum idle runner count of ```1```. This means that this pool will create by default one runner, and will automatically add more, as jobs are triggered on github. The idea is to have at least one runner ready to accept a workflow job. The pool will keep adding workers until the max runner count is reached. Once a workflow job is complete, the runner is automatically deleted, and replaced.
To update the pool, we cam use the following command: To update the pool, we cam use the following command:
```bash ```bash
ubuntu@experiments:~$ garm-cli pool update fb25f308-7ad2-4769-988e-6ec2935f642a --enabled=true ubuntu@experiments:~$ garm-cli pool update fb25f308-7ad2-4769-988e-6ec2935f642a --enabled=true
+------------------+-------------------------------------------------------------+ +------------------+-------------------------------------------------------------+
| FIELD | VALUE | | FIELD | VALUE |
+------------------+-------------------------------------------------------------+ +------------------+-------------------------------------------------------------+
| ID | fb25f308-7ad2-4769-988e-6ec2935f642a | | ID | fb25f308-7ad2-4769-988e-6ec2935f642a |
| Provider Name | lxd_local | | Provider Name | lxd_local |
| Image | ubuntu:20.04 | | Image | ubuntu:20.04 |
| Flavor | default | | Flavor | default |
| OS Type | linux | | OS Type | linux |
| OS Architecture | amd64 | | OS Architecture | amd64 |
| Max Runners | 5 | | Max Runners | 5 |
| Min Idle Runners | 1 | | Min Idle Runners | 1 |
| Tags | ubuntu, simple-runner, repo-runner, self-hosted, x64, linux | | Tags | ubuntu, simple-runner, repo-runner, self-hosted, x64, linux |
| Belongs to | gabriel-samfira/scripts | | Belongs to | gabriel-samfira/scripts |
| Level | repo | | Level | repo |
| Enabled | true | | Enabled | true |
+------------------+-------------------------------------------------------------+ +------------------+-------------------------------------------------------------+
``` ```
Now, if we list the runners, we should see one being created: Now, if we list the runners, we should see one being created:
```bash ```bash
ubuntu@experiments:~$ garm-cli runner ls fb25f308-7ad2-4769-988e-6ec2935f642a ubuntu@experiments:~$ garm-cli runner ls fb25f308-7ad2-4769-988e-6ec2935f642a
+-------------------------------------------+----------------+---------------+--------------------------------------+ +-------------------------------------------+----------------+---------------+--------------------------------------+
| NAME | STATUS | RUNNER STATUS | POOL ID | | NAME | STATUS | RUNNER STATUS | POOL ID |
+-------------------------------------------+----------------+---------------+--------------------------------------+ +-------------------------------------------+----------------+---------------+--------------------------------------+
| garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | pending_create | pending | fb25f308-7ad2-4769-988e-6ec2935f642a | | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | pending_create | pending | fb25f308-7ad2-4769-988e-6ec2935f642a |
+-------------------------------------------+----------------+---------------+--------------------------------------+ +-------------------------------------------+----------------+---------------+--------------------------------------+
``` ```
We can also do a show on that runner to get more info: We can also do a show on that runner to get more info:
```bash ```bash
ubuntu@experiments:~$ garm-cli runner show garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe ubuntu@experiments:~$ garm-cli runner show garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe
+-----------------+-------------------------------------------+ +-----------------+-------------------------------------------+
| FIELD | VALUE | | FIELD | VALUE |
+-----------------+-------------------------------------------+ +-----------------+-------------------------------------------+
| ID | 089d63c9-5567-4318-a3a6-e065685c975b | | ID | 089d63c9-5567-4318-a3a6-e065685c975b |
| Provider ID | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | | Provider ID | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe |
| Name | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | | Name | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe |
| OS Type | linux | | OS Type | linux |
| OS Architecture | amd64 | | OS Architecture | amd64 |
| OS Name | ubuntu | | OS Name | ubuntu |
| OS Version | focal | | OS Version | focal |
| Status | running | | Status | running |
| Runner Status | pending | | Runner Status | pending |
| Pool ID | fb25f308-7ad2-4769-988e-6ec2935f642a | | Pool ID | fb25f308-7ad2-4769-988e-6ec2935f642a |
+-----------------+-------------------------------------------+ +-----------------+-------------------------------------------+
``` ```
If we check out LXD, we can see the instance was created and is currently being bootstrapped: If we check out LXD, we can see the instance was created and is currently being bootstrapped:
```bash ```bash
ubuntu@experiments:~$ lxc list ubuntu@experiments:~$ lxc list
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+ +-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+ +-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
| garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | RUNNING | 10.247.246.219 (enp5s0) | | VIRTUAL-MACHINE | 0 | | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | RUNNING | 10.247.246.219 (enp5s0) | | VIRTUAL-MACHINE | 0 |
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+ +-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
``` ```
It might take a couple of minutes for the runner to come online, as the instance will do a full upgrade, then download the runner and install it. But once the installation is done you should see something like this: It might take a couple of minutes for the runner to come online, as the instance will do a full upgrade, then download the runner and install it. But once the installation is done you should see something like this:
```bash ```bash
ubuntu@experiments:~$ garm-cli runner show garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe ubuntu@experiments:~$ garm-cli runner show garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| FIELD | VALUE | | FIELD | VALUE |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| ID | 089d63c9-5567-4318-a3a6-e065685c975b | | ID | 089d63c9-5567-4318-a3a6-e065685c975b |
| Provider ID | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | | Provider ID | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe |
| Name | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | | Name | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe |
| OS Type | linux | | OS Type | linux |
| OS Architecture | amd64 | | OS Architecture | amd64 |
| OS Name | ubuntu | | OS Name | ubuntu |
| OS Version | focal | | OS Version | focal |
| Status | running | | Status | running |
| Runner Status | idle | | Runner Status | idle |
| Pool ID | fb25f308-7ad2-4769-988e-6ec2935f642a | | Pool ID | fb25f308-7ad2-4769-988e-6ec2935f642a |
| Status Updates | 2022-05-06T13:21:54: downloading tools from https://github.com/actions/runner/releases/download/v2.291.1/actions-runner-linux-x64-2.291.1.tar.gz | | Status Updates | 2022-05-06T13:21:54: downloading tools from https://github.com/actions/runner/releases/download/v2.291.1/actions-runner-linux-x64-2.291.1.tar.gz |
| | 2022-05-06T13:21:56: extracting runner | | | 2022-05-06T13:21:56: extracting runner |
| | 2022-05-06T13:21:58: installing dependencies | | | 2022-05-06T13:21:58: installing dependencies |
| | 2022-05-06T13:22:07: configuring runner | | | 2022-05-06T13:22:07: configuring runner |
| | 2022-05-06T13:22:12: installing runner service | | | 2022-05-06T13:22:12: installing runner service |
| | 2022-05-06T13:22:12: starting service | | | 2022-05-06T13:22:12: starting service |
| | 2022-05-06T13:22:13: runner successfully installed | | | 2022-05-06T13:22:13: runner successfully installed |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
``` ```
If we list the runners for this pool, we should see one runner with a ```RUNNER STATUS``` of ```idle```: If we list the runners for this pool, we should see one runner with a ```RUNNER STATUS``` of ```idle```:
```bash ```bash
ubuntu@experiments:~$ garm-cli runner ls fb25f308-7ad2-4769-988e-6ec2935f642a ubuntu@experiments:~$ garm-cli runner ls fb25f308-7ad2-4769-988e-6ec2935f642a
+-------------------------------------------+---------+---------------+--------------------------------------+ +-------------------------------------------+---------+---------------+--------------------------------------+
| NAME | STATUS | RUNNER STATUS | POOL ID | | NAME | STATUS | RUNNER STATUS | POOL ID |
+-------------------------------------------+---------+---------------+--------------------------------------+ +-------------------------------------------+---------+---------------+--------------------------------------+
| garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | running | idle | fb25f308-7ad2-4769-988e-6ec2935f642a | | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | running | idle | fb25f308-7ad2-4769-988e-6ec2935f642a |
+-------------------------------------------+---------+---------------+--------------------------------------+ +-------------------------------------------+---------+---------------+--------------------------------------+
``` ```
## Updating a pool ## Updating a pool
Let's update the pool and request that it maintain a number of minimum idle runners equal to 3: Let's update the pool and request that it maintain a number of minimum idle runners equal to 3:
```bash ```bash
ubuntu@experiments:~$ garm-cli pool update fb25f308-7ad2-4769-988e-6ec2935f642a \ ubuntu@experiments:~$ garm-cli pool update fb25f308-7ad2-4769-988e-6ec2935f642a \
--min-idle-runners=3 \ --min-idle-runners=3 \
--max-runners=10 --max-runners=10
+------------------+----------------------------------------------------------------------------------+ +------------------+----------------------------------------------------------------------------------+
| FIELD | VALUE | | FIELD | VALUE |
+------------------+----------------------------------------------------------------------------------+ +------------------+----------------------------------------------------------------------------------+
| ID | fb25f308-7ad2-4769-988e-6ec2935f642a | | ID | fb25f308-7ad2-4769-988e-6ec2935f642a |
| Provider Name | lxd_local | | Provider Name | lxd_local |
| Image | ubuntu:20.04 | | Image | ubuntu:20.04 |
| Flavor | default | | Flavor | default |
| OS Type | linux | | OS Type | linux |
| OS Architecture | amd64 | | OS Architecture | amd64 |
| Max Runners | 10 | | Max Runners | 10 |
| Min Idle Runners | 3 | | Min Idle Runners | 3 |
| Tags | ubuntu, simple-runner, repo-runner, self-hosted, x64, linux | | Tags | ubuntu, simple-runner, repo-runner, self-hosted, x64, linux |
| Belongs to | gabriel-samfira/scripts | | Belongs to | gabriel-samfira/scripts |
| Level | repo | | Level | repo |
| Enabled | true | | Enabled | true |
| Instances | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe (089d63c9-5567-4318-a3a6-e065685c975b) | | Instances | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe (089d63c9-5567-4318-a3a6-e065685c975b) |
+------------------+----------------------------------------------------------------------------------+ +------------------+----------------------------------------------------------------------------------+
``` ```
Now if we list runners we should see 2 more in ```pending``` state: Now if we list runners we should see 2 more in ```pending``` state:
```bash ```bash
ubuntu@experiments:~$ garm-cli runner ls fb25f308-7ad2-4769-988e-6ec2935f642a ubuntu@experiments:~$ garm-cli runner ls fb25f308-7ad2-4769-988e-6ec2935f642a
+-------------------------------------------+---------+---------------+--------------------------------------+ +-------------------------------------------+---------+---------------+--------------------------------------+
| NAME | STATUS | RUNNER STATUS | POOL ID | | NAME | STATUS | RUNNER STATUS | POOL ID |
+-------------------------------------------+---------+---------------+--------------------------------------+ +-------------------------------------------+---------+---------------+--------------------------------------+
| garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | running | idle | fb25f308-7ad2-4769-988e-6ec2935f642a | | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | running | idle | fb25f308-7ad2-4769-988e-6ec2935f642a |
+-------------------------------------------+---------+---------------+--------------------------------------+ +-------------------------------------------+---------+---------------+--------------------------------------+
| garm-bc180c6c-6e31-4c7b-8ce1-da0ffd76e247 | running | pending | fb25f308-7ad2-4769-988e-6ec2935f642a | | garm-bc180c6c-6e31-4c7b-8ce1-da0ffd76e247 | running | pending | fb25f308-7ad2-4769-988e-6ec2935f642a |
+-------------------------------------------+---------+---------------+--------------------------------------+ +-------------------------------------------+---------+---------------+--------------------------------------+
| garm-37c5daf4-18c5-47fc-95de-8c1656889093 | running | pending | fb25f308-7ad2-4769-988e-6ec2935f642a | | garm-37c5daf4-18c5-47fc-95de-8c1656889093 | running | pending | fb25f308-7ad2-4769-988e-6ec2935f642a |
+-------------------------------------------+---------+---------------+--------------------------------------+ +-------------------------------------------+---------+---------------+--------------------------------------+
``` ```
We can see them in LXC as well: We can see them in LXC as well:
```bash ```bash
ubuntu@experiments:~$ lxc list ubuntu@experiments:~$ lxc list
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+ +-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+ +-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
| garm-37c5daf4-18c5-47fc-95de-8c1656889093 | RUNNING | | | VIRTUAL-MACHINE | 0 | | garm-37c5daf4-18c5-47fc-95de-8c1656889093 | RUNNING | | | VIRTUAL-MACHINE | 0 |
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+ +-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
| garm-bc180c6c-6e31-4c7b-8ce1-da0ffd76e247 | RUNNING | | | VIRTUAL-MACHINE | 0 | | garm-bc180c6c-6e31-4c7b-8ce1-da0ffd76e247 | RUNNING | | | VIRTUAL-MACHINE | 0 |
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+ +-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
| garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | RUNNING | 10.247.246.219 (enp5s0) | | VIRTUAL-MACHINE | 0 | | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | RUNNING | 10.247.246.219 (enp5s0) | | VIRTUAL-MACHINE | 0 |
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+ +-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
``` ```
Once they transition to ```idle```, you should see them in your repo settings, under ```Actions --> Runners```. Once they transition to ```idle```, you should see them in your repo settings, under ```Actions --> Runners```.
The procedure is identical for organizations. Have a look at the garm-cli help: The procedure is identical for organizations. Have a look at the garm-cli help:
```bash
ubuntu@experiments:~$ garm-cli -h
CLI for the github self hosted runners manager.
```bash Usage:
ubuntu@experiments:~$ garm-cli -h garm-cli [command]
CLI for the github self hosted runners manager.
Usage: Available Commands:
garm-cli [command] completion Generate the autocompletion script for the specified shell
credentials List configured credentials
debug-log Stream garm log
enterprise Manage enterprise
help Help about any command
init Initialize a newly installed garm
organization Manage organizations
pool List pools
profile Add, delete or update profiles
provider Interacts with the providers API resource.
repository Manage repositories
runner List runners in a pool
version Print version and exit
Available Commands: Flags:
completion Generate the autocompletion script for the specified shell --debug Enable debug on all API calls
credentials List configured credentials -h, --help help for garm-cli
debug-log Stream garm log
enterprise Manage enterprise
help Help about any command
init Initialize a newly installed garm
organization Manage organizations
pool List pools
profile Add, delete or update profiles
provider Interacts with the providers API resource.
repository Manage repositories
runner List runners in a pool
version Print version and exit
Flags: Use "garm-cli [command] --help" for more information about a command.
--debug Enable debug on all API calls
-h, --help help for garm-cli
Use "garm-cli [command] --help" for more information about a command. ```
```

103
doc/webhooks_and_callbacks.md
View file

@ -1,18 +1,18 @@
## Webhooks # Webhooks
Garm is designed to auto-scale github runners based on a few simple rules: Garm is designed to auto-scale github runners based on a few simple rules:
* A minimum idle runner count can be set for a pool. Garm will attempt to maintain that minimum of idle runners, ready to be used by your workflows. * A minimum idle runner count can be set for a pool. Garm will attempt to maintain that minimum of idle runners, ready to be used by your workflows.
* A maximum number of runners for a pool. This is a hard limit of runners a pool will create, regardless of minimum idle runners. * A maximum number of runners for a pool. This is a hard limit of runners a pool will create, regardless of minimum idle runners.
* When a runner is scheduled by github, ```garm``` will automatically spin up a new runner to replace it, obeying the maximum hard limit defined. * When a runner is scheduled by github, ```garm``` will automatically spin up a new runner to replace it, obeying the maximum hard limit defined.
To achieve this, ```garm``` relies on [GitHub Webhooks](https://docs.github.com/en/developers/webhooks-and-events/webhooks/about-webhooks). Webhooks allow ```garm``` to react to workflow events from your repository or organization. To achieve this, ```garm``` relies on [GitHub Webhooks](https://docs.github.com/en/developers/webhooks-and-events/webhooks/about-webhooks). Webhooks allow ```garm``` to react to workflow events from your repository or organization.
In your repository or organization, navigate to ```Settings --> Webhooks```. In the ```Payload URL``` field, enter the URL to the ```garm``` webhook endpoint. The ```garm``` API endpoint for webhooks is: In your repository or organization, navigate to ```Settings --> Webhooks```. In the ```Payload URL``` field, enter the URL to the ```garm``` webhook endpoint. The ```garm``` API endpoint for webhooks is:
``` ```txt
POST /webhooks POST /webhooks
``` ```
If ```garm``` is running on a server under the domain ```garm.example.com```, then that field should be set to ```https://garm.example.com/webhooks```. If ```garm``` is running on a server under the domain ```garm.example.com```, then that field should be set to ```https://garm.example.com/webhooks```.
@ -20,61 +20,60 @@ In the webhook configuration page under ```Content type``` you will need to sele
The webhook secret must be secure. Use something like this to generate one: The webhook secret must be secure. Use something like this to generate one:
```bash ```bash
gabriel@rossak:~$ function generate_secret () { gabriel@rossak:~$ function generate_secret () {
tr -dc 'a-zA-Z0-9!@#$%^&*()_+?><~\`;' < /dev/urandom | head -c 64; tr -dc 'a-zA-Z0-9!@#$%^&*()_+?><~\`;' < /dev/urandom | head -c 64;
echo '' echo ''
} }
gabriel@rossak:~$ generate_secret gabriel@rossak:~$ generate_secret
9Q<fVm5dtRhUIJ>*nsr*S54g0imK64(!2$Ns6C!~VsH(p)cFj+AMLug%LM!R%FOQ 9Q<fVm5dtRhUIJ>*nsr*S54g0imK64(!2$Ns6C!~VsH(p)cFj+AMLug%LM!R%FOQ
``` ```
Next, you can choose which events GitHub should send to ```garm``` via webhooks. Click on ```Let me select individual events``` and select ```Workflow jobs``` (should be at the bottom). You can send everything if you want, but any events ```garm``` doesn't care about will simply be ignored. Next, you can choose which events GitHub should send to ```garm``` via webhooks. Click on ```Let me select individual events``` and select ```Workflow jobs``` (should be at the bottom). You can send everything if you want, but any events ```garm``` doesn't care about will simply be ignored.
## The callback_url option ## The callback_url option
Your runners will call back home with status updates as they install. Once they are set up, they will also send the GitHub agent ID they were allocated. You will need to configure the ```callback_url``` option in the ```garm``` server config. This URL needs to point to the following API endpoint: Your runners will call back home with status updates as they install. Once they are set up, they will also send the GitHub agent ID they were allocated. You will need to configure the ```callback_url``` option in the ```garm``` server config. This URL needs to point to the following API endpoint:
``` ```txt
POST /api/v1/callbacks/status POST /api/v1/callbacks/status
``` ```
Example of a runner sending status updates: Example of a runner sending status updates:
```bash ```bash
garm-cli runner show garm-f5227755-129d-4e2d-b306-377a8f3a5dfe garm-cli runner show garm-f5227755-129d-4e2d-b306-377a8f3a5dfe
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| FIELD | VALUE | | FIELD | VALUE |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| ID | 1afb407b-e9f7-4d75-a410-fc4a8c2dbe6c | | ID | 1afb407b-e9f7-4d75-a410-fc4a8c2dbe6c |
| Provider ID | garm-f5227755-129d-4e2d-b306-377a8f3a5dfe | | Provider ID | garm-f5227755-129d-4e2d-b306-377a8f3a5dfe |
| Name | garm-f5227755-129d-4e2d-b306-377a8f3a5dfe | | Name | garm-f5227755-129d-4e2d-b306-377a8f3a5dfe |
| OS Type | linux | | OS Type | linux |
| OS Architecture | amd64 | | OS Architecture | amd64 |
| OS Name | ubuntu | | OS Name | ubuntu |
| OS Version | focal | | OS Version | focal |
| Status | running | | Status | running |
| Runner Status | idle | | Runner Status | idle |
| Pool ID | 98f438b9-5549-4eaf-9bb7-1781533a455d | | Pool ID | 98f438b9-5549-4eaf-9bb7-1781533a455d |
| Status Updates | 2022-05-05T11:32:41: downloading tools from https://github.com/actions/runner/releases/download/v2.290.1/actions-runner-linux-x64-2.290.1.tar.gz | | Status Updates | 2022-05-05T11:32:41: downloading tools from https://github.com/actions/runner/releases/download/v2.290.1/actions-runner-linux-x64-2.290.1.tar.gz |
| | 2022-05-05T11:32:43: extracting runner | | | 2022-05-05T11:32:43: extracting runner |
| | 2022-05-05T11:32:47: installing dependencies | | | 2022-05-05T11:32:47: installing dependencies |
| | 2022-05-05T11:32:55: configuring runner | | | 2022-05-05T11:32:55: configuring runner |
| | 2022-05-05T11:32:59: installing runner service | | | 2022-05-05T11:32:59: installing runner service |
| | 2022-05-05T11:33:00: starting service | | | 2022-05-05T11:33:00: starting service |
| | 2022-05-05T11:33:00: runner successfully installed | | | 2022-05-05T11:33:00: runner successfully installed |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
``` ```
This URL must be set and must be accessible by the instance. If you wish to restrict access to it, a reverse proxy can be configured to accept requests only from networks in which the runners ```garm``` manages will be spun up. This URL doesn't need to be globally accessible, it just needs to be accessible by the instances. This URL must be set and must be accessible by the instance. If you wish to restrict access to it, a reverse proxy can be configured to accept requests only from networks in which the runners ```garm``` manages will be spun up. This URL doesn't need to be globally accessible, it just needs to be accessible by the instances.
For example, in a scenario where you expose the API endpoint directly, this setting could look like the following: For example, in a scenario where you expose the API endpoint directly, this setting could look like the following:
```toml ```toml
callback_url = "https://garm.example.com/api/v1/callbacks/status" callback_url = "https://garm.example.com/api/v1/callbacks/status"
``` ```
Authentication is done using a short-lived JWT token, that gets generated for a particular instance that we are spinning up. That JWT token grants access to the instance to only update it's own status and to fetch metadata for itself. No other API endpoints will work with that JWT token. The validity of the token is equal to the pool bootstrap timeout value (default 20 minutes) plus the garm polling interval (5 minutes). Authentication is done using a short-lived JWT token, that gets generated for a particular instance that we are spinning up. That JWT token grants access to the instance to only update it's own status and to fetch metadata for itself. No other API endpoints will work with that JWT token. The validity of the token is equal to the pool bootstrap timeout value (default 20 minutes) plus the garm polling interval (5 minutes).
@ -84,12 +83,12 @@ There is a sample ```nginx``` config [in the testdata folder](/testdata/nginx-se
The metadata URL is the base URL for any information an instance may need to fetch in order to finish setting itself up. As this URL may be placed behind a reverse proxy, you'll need to configure it in the ```garm``` config file. Ultimately this URL will need to point to the following ```garm``` API endpoint: The metadata URL is the base URL for any information an instance may need to fetch in order to finish setting itself up. As this URL may be placed behind a reverse proxy, you'll need to configure it in the ```garm``` config file. Ultimately this URL will need to point to the following ```garm``` API endpoint:
```bash ```bash
GET /api/v1/metadata GET /api/v1/metadata
``` ```
This URL needs to be accessible only by the instances ```garm``` sets up. This URL will not be used by anyone else. To configure it in ```garm``` add the following line in the ```[default]``` section of your ```garm``` config: This URL needs to be accessible only by the instances ```garm``` sets up. This URL will not be used by anyone else. To configure it in ```garm``` add the following line in the ```[default]``` section of your ```garm``` config:
```toml ```toml
metadata_url = "https://garm.example.com/api/v1/metadata" metadata_url = "https://garm.example.com/api/v1/metadata"
``` ```