DevFW-CICD/garm
15
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

Merge pull request #129 from gabriel-samfira/docs-update

Browse source 
Update some docs
This commit is contained in:
Gabriel 2023-07-20 13:52:34 +03:00 • committed by GitHub
commit 0987bba343
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
25 changed files with 1014 additions and 748 deletions
Download patch file Download diff file Expand all files Collapse all files

127
README.md
View file

@ -1,14 +1,14 @@
# GitHub Actions Runner Manager (garm)
# GitHub Actions Runner Manager (GARM)
[![Go Tests](https://github.com/cloudbase/garm/actions/workflows/go-tests.yml/badge.svg)](https://github.com/cloudbase/garm/actions/workflows/go-tests.yml)
Welcome to garm!
Welcome to GARM!
Garm enables you to create and automatically maintain pools of [self-hosted GitHub runners](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners), with autoscaling that can be used inside your github workflow runs.
The goal of ```garm``` is to be simple to set up, simple to configure and simple to use. It is a single binary that can run on any GNU/Linux machine without any other requirements other than the providers it creates the runners in. It is intended to be easy to deploy in any environment and can create runners in any system you can write a provider for. There is no complicated setup process and no extremely complex concepts to understand. Once set up, it's meant to stay out of your way.
The goal of ```GARM``` is to be simple to set up, simple to configure and simple to use. It is a single binary that can run on any GNU/Linux machine without any other requirements other than the providers it creates the runners in. It is intended to be easy to deploy in any environment and can create runners in any system you can write a provider for. There is no complicated setup process and no extremely complex concepts to understand. Once set up, it's meant to stay out of your way.
Garm supports creating pools on either GitHub itself or on your own deployment of [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@3.5/admin/overview/about-github-enterprise-server). For instructions on how to use ```garm``` with GHE, see the [credentials](/doc/github_credentials.md) section of the documentation.
Garm supports creating pools on either GitHub itself or on your own deployment of [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@3.5/admin/overview/about-github-enterprise-server). For instructions on how to use ```GARM``` with GHE, see the [credentials](/doc/github_credentials.md) section of the documentation.
## Join us on slack
@ -18,121 +18,36 @@ Whether you're running into issues or just want to drop by and say "hi", feel fr
## Installing
## Build from source
Check out the [quickstart](/doc/quickstart.md) document for instructions on how to install ```GARM```. If you'd like to build from source, check out the [building from source](/doc/building_from_source.md) document.
You need to have Go installed, then run:
## Installing external providers
```bash
git clone https://github.com/cloudbase/garm
cd garm
go install ./...
```
External providers are binaries that GARM calls into to create runners in a particular IaaS. There are currently two external providers available:
You should now have both ```garm``` and ```garm-cli``` in your ```$GOPATH/bin``` folder.
* [OpenStack](https://github.com/cloudbase/garm-provider-openstack)
* [Azure](https://github.com/cloudbase/garm-provider-azure)
If you have docker/podman installed, you can also build statically linked binaries by running:
```bash
make build-static
```
The ```garm``` and ```garm-cli``` binaries will be built and copied to the ```bin/``` folder in your current working directory.
## Install the service
Add a new system user:
```bash
useradd --shell /usr/bin/false \
--system \
--groups lxd \
--no-create-home garm
```
The ```lxd``` group is only needed if you have a local LXD install and want to connect to the unix socket to use it. If you're connecting to a remote LXD server over TCP, you can skip adding the ```garm``` user to the ```lxd``` group.
Copy the binary to somewhere in the system ```$PATH```:
```bash
sudo cp $(go env GOPATH)/bin/garm /usr/local/bin/garm
```
Or if you built garm using ```make```:
```bash
sudo cp ./bin/garm /usr/local/bin/garm
```
Create the config folder:
```bash
sudo mkdir -p /etc/garm
```
Copy the config template:
```bash
sudo cp ./testdata/config.toml /etc/garm/
```
Copy the external provider (optional):
```bash
sudo cp -a ./contrib/providers.d /etc/garm/
```
Copy the systemd service file:
```bash
sudo cp ./contrib/garm.service /etc/systemd/system/
```
Change permissions on config folder:
```bash
sudo chown -R garm:garm /etc/garm
sudo chmod 750 -R /etc/garm
```
Enable the service:
```bash
sudo systemctl enable garm
```
Customize the config in ```/etc/garm/config.toml```, and start the service:
```bash
sudo systemctl start garm
```
Follow the instructions in the README of each provider to install them.
## Configuration
The ```garm``` configuration is a simple ```toml```. A sample of the config file can be found in [the testdata folder](/testdata/config.toml).
The ```GARM``` configuration is a simple ```toml```. The sample config file in [the testdata folder](/testdata/config.toml) is fairly well commented and should be enough to get you started. The configuration file is split into several sections, each of which is documented in its own page. The sections are:
There are 3 major sections of the config that require your attention:
* [The default section](/doc/config_default.md)
* [Database](/doc/database.md)
* [Github credentials](/doc/github_credentials.md)
* [Providers](/doc/providers.md)
* [Metrics](/doc/config_metrics.md)
* [JWT authentication](/doc/config_jwt_auth.md)
* [API server](/doc/config_api_server.md)
* [Github credentials section](/doc/github_credentials.md)
* [Providers section](/doc/providers.md)
* [The database section](/doc/database.md)
## Optimizing your runners
Once you've configured your database, providers and github credentials, you'll need to configure your [webhooks and the callback_url](/doc/webhooks_and_callbacks.md).
At this point, you should be done. Have a look at the [running garm document](/doc/running_garm.md) for usage instructions and available features.
If you would like to use ```garm``` with a different IaaS than the ones already available, have a look at the [writing an external provider](/doc/external_provider.md) page.
If you like to optimize the startup time of new instance, take a look at the [performance considerations](/doc/performance_considerations.md) page.
## Security considerations
Garm does not apply any ACLs of any kind to the instances it creates. That task remains in the responsibility of the user. [Here is a guide for creating ACLs in LXD](https://linuxcontainers.org/lxd/docs/master/howto/network_acls/). You can of course use ```iptables``` or ```nftables``` to create any rules you wish. I recommend you create a separate isolated lxd bridge for runners, and secure it using ACLs/iptables/nftables.
You must make sure that the code that runs as part of the workflows is trusted, and if that cannot be done, you must make sure that any malicious code that will be pulled in by the actions and run as part of a workload, is as contained as possible. There is a nice article about [securing your workflow runs here](https://blog.gitguardian.com/github-actions-security-cheat-sheet/).
If you would like to optimize the startup time of new instance, take a look at the [performance considerations](/doc/performance_considerations.md) page.
## Write your own provider
The providers are interfaces between ```garm``` and a particular IaaS in which we spin up GitHub Runners. These providers can be either **native** or **external**. The **native** providers are written in ```Go```, and must implement [the interface defined here](https://github.com/cloudbase/garm/blob/main/runner/common/provider.go#L22-L39). **External** providers can be written in any language, as they are in the form of an external executable that ```garm``` calls into.
The providers are interfaces between ```GARM``` and a particular IaaS in which we spin up GitHub Runners. These providers can be either **native** or **external**. The **native** providers are written in ```Go```, and must implement [the interface defined here](https://github.com/cloudbase/garm/blob/main/runner/common/provider.go#L22-L39). **External** providers can be written in any language, as they are in the form of an external executable that ```GARM``` calls into.
There is currently one **native** provider for [LXD](https://linuxcontainers.org/lxd/) and two **external** providers for [Openstack and Azure](/contrib/providers.d/).

8
cloudconfig/templates.go
View file

@ -40,6 +40,10 @@ GITHUB_TOKEN=$(curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X G
function call() {
PAYLOAD="$1"
[[ $CALLBACK_URL =~ ^(.*)/status$ ]]
if [ -z "$BASH_REMATCH" ];then
CALLBACK_URL="${CALLBACK_URL}/status"
fi
curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d "${PAYLOAD}" -H 'Accept: application/json' -H "Authorization: Bearer ${BEARER_TOKEN}" "${CALLBACK_URL}" || echo "failed to call home: exit code ($?)"
}
@ -350,6 +354,10 @@ $GHRunnerGroup = "{{.GitHubRunnerGroup}}"
function Install-Runner() {
$CallbackURL="{{.CallbackURL}}"
if (!$CallbackURL.EndsWith("/status")) {
$CallbackURL = "$CallbackURL/status"
}
if ($Token.Length -eq 0) {
Throw "missing callback authentication token"
}

15
config/config.go
View file

@ -47,9 +47,6 @@ func NewConfig(cfgFile string) (*Config, error) {
if _, err := toml.DecodeFile(cfgFile, &config); err != nil {
return nil, errors.Wrap(err, "decoding toml")
}
if config.Default.ConfigDir == "" {
config.Default.ConfigDir = appdefaults.DefaultConfigDir
}
if err := config.Validate(); err != nil {
return nil, errors.Wrap(err, "validating config")
}
@ -108,10 +105,6 @@ func (c *Config) Validate() error {
}
type Default struct {
// ConfigDir is the folder where the runner may save any aditional files
// or configurations it may need. Things like auto-generated SSH keys that
// may be used to access the runner instances.
ConfigDir string `toml:"config_dir,omitempty" json:"config-dir,omitempty"`
// CallbackURL is the URL where the instances can send back status reports.
CallbackURL string `toml:"callback_url" json:"callback-url"`
// MetadataURL is the URL where instances can fetch information they may need
@ -139,14 +132,6 @@ func (d *Default) Validate() error {
return errors.Wrap(err, "validating metadata_url")
}
if d.ConfigDir == "" {
return fmt.Errorf("config_dir cannot be empty")
}
if _, err := os.Stat(d.ConfigDir); err != nil {
return errors.Wrap(err, "accessing config dir")
}
return nil
}

47
config/config_test.go
View file

@ -32,7 +32,6 @@ var (
func getDefaultSectionConfig(configDir string) Default {
return Default{
ConfigDir: configDir,
CallbackURL: "https://garm.example.com/",
MetadataURL: "https://garm.example.com/api/v1/metadata",
LogFile: filepath.Join(configDir, "garm.log"),
@ -152,7 +151,6 @@ func TestDefaultSectionConfig(t *testing.T) {
cfg: Default{
CallbackURL: "",
MetadataURL: cfg.MetadataURL,
ConfigDir: cfg.ConfigDir,
},
errString: "missing callback_url",
},
@ -161,28 +159,9 @@ func TestDefaultSectionConfig(t *testing.T) {
cfg: Default{
CallbackURL: cfg.CallbackURL,
MetadataURL: "",
ConfigDir: cfg.ConfigDir,
},
errString: "missing metadata-url",
},
{
name: "ConfigDir cannot be empty",
cfg: Default{
CallbackURL: cfg.CallbackURL,
MetadataURL: cfg.MetadataURL,
ConfigDir: "",
},
errString: "config_dir cannot be empty",
},
{
name: "config_dir must exist and be accessible",
cfg: Default{
CallbackURL: cfg.CallbackURL,
MetadataURL: cfg.MetadataURL,
ConfigDir: "/i/do/not/exist",
},
errString: "accessing config dir: stat /i/do/not/exist:.*",
},
}
for _, tc := range tests {
@ -560,7 +539,6 @@ func TestNewConfig(t *testing.T) {
require.Nil(t, err)
require.NotNil(t, cfg)
require.Equal(t, "https://garm.example.com/", cfg.Default.CallbackURL)
require.Equal(t, "./testdata", cfg.Default.ConfigDir)
require.Equal(t, "0.0.0.0", cfg.APIServer.Bind)
require.Equal(t, 9998, cfg.APIServer.Port)
require.Equal(t, false, cfg.APIServer.UseTLS)
@ -574,31 +552,6 @@ func TestNewConfig(t *testing.T) {
require.Equal(t, timeToLive("48h"), cfg.JWTAuth.TimeToLive)
}
func TestNewConfigEmptyConfigDir(t *testing.T) {
dirPath, err := os.MkdirTemp("", "garm-config-test")
if err != nil {
t.Fatalf("failed to create temporary directory: %s", err)
}
defer os.RemoveAll(dirPath)
appdefaults.DefaultConfigDir = dirPath
cfg, err := NewConfig("testdata/test-empty-config-dir.toml")
require.Nil(t, err)
require.NotNil(t, cfg)
require.Equal(t, cfg.Default.ConfigDir, dirPath)
require.Equal(t, "https://garm.example.com/", cfg.Default.CallbackURL)
require.Equal(t, "0.0.0.0", cfg.APIServer.Bind)
require.Equal(t, 9998, cfg.APIServer.Port)
require.Equal(t, false, cfg.APIServer.UseTLS)
require.Equal(t, DBBackendType("mysql"), cfg.Database.DbBackend)
require.Equal(t, "test", cfg.Database.MySQL.Username)
require.Equal(t, "test", cfg.Database.MySQL.Password)
require.Equal(t, "127.0.0.1", cfg.Database.MySQL.Hostname)
require.Equal(t, "garm", cfg.Database.MySQL.DatabaseName)
require.Equal(t, "bocyasicgatEtenOubwonIbsudNutDom", cfg.JWTAuth.Secret)
require.Equal(t, timeToLive("48h"), cfg.JWTAuth.TimeToLive)
}
func TestNewConfigInvalidTomlPath(t *testing.T) {
cfg, err := NewConfig("this is not a file path")
require.Nil(t, cfg)

25
doc/building_from_source.md Normal file
View file

@ -0,0 +1,25 @@
# Building GARM from source
The procedure is simple. You will need to gave [go](https://golang.org/) installed as well as `make`.
First, clone the repository:
```bash
git clone https://github.com/cloudbase/garm
```
Then build garm:
```bash
make
```
You should now have both `garm` and `garm-cli` available in the `./bin` folder.
If you have docker/podman installed, you can also build a static binary against `musl`:
```bash
make build-static
```
This command will also build for both AMD64 and ARM64. Resulting binaries will be in the `./bin` folder.

34
doc/config_api_server.md Normal file
View file

@ -0,0 +1,34 @@
# The API server config section
This section allows you to configure the GARM API server. The API server is responsible for serving all the API endpoints used by the `garm-cli`, the runners that phone home their status and by GitHub when it sends us webhooks.
The config options are fairly straight forward.
```toml
[apiserver]
# Bind the API to this IP
bind = "0.0.0.0"
# Bind the API to this port
port = 9997
# Whether or not to set up TLS for the API endpoint. If this is set to true,
# you must have a valid apiserver.tls section.
use_tls = false
# Set a list of allowed origins
# By default, if this option is ommited or empty, we will check
# only that the origin is the same as the originating server.
# A literal of "*" will allow any origin
cors_origins = ["*"]
[apiserver.tls]
# Path on disk to a x509 certificate bundle.
# NOTE: if your certificate is signed by an intermediary CA, this file
# must contain the entire certificate bundle needed for clients to validate
# the certificate. This usually means concatenating the certificate and the
# CA bundle you received.
certificate = ""
# The path on disk to the corresponding private key for the certificate.
key = ""
```
The GARM API server has the option to enable TLS, but I suggest you use a reverse proxy and enable TLS termination in that reverse proxy. There is an `nginx` sample in this repository with TLS termination enabled.
You can of course enable TLS in both garm and the reverse proxy. The choice is yours.

169
doc/config_default.md Normal file
View file

@ -0,0 +1,169 @@
# The default config section
The `default` config section holds configuration options that don't need a category of their own, but are essential to the operation of the service. In this section we will detail each of the options available in the `default` section.
```toml
[default]
# This URL is used by instances to send back status messages as they install
# the github actions runner. Status messages can be seen by querying the
# runner status in garm.
# Note: If you're using a reverse proxy in front of your garm installation,
# this URL needs to point to the address of the reverse proxy. Using TLS is
# highly encouraged.
callback_url = "https://garm.example.com/api/v1/callbacks/status"
# This URL is used by instances to retrieve information they need to set themselves
# up. Access to this URL is granted using the same JWT token used to send back
# status updates. Once the instance transitions to "installed" or "failed" state,
# access to both the status and metadata endpoints is disabled.
# Note: If you're using a reverse proxy in front of your garm installation,
# this URL needs to point to the address of the reverse proxy. Using TLS is
# highly encouraged.
metadata_url = "https://garm.example.com/api/v1/metadata"
# Uncomment this line if you'd like to log to a file instead of standard output.
# log_file = "/tmp/runner-manager.log"
# Enable streaming logs via web sockets. Use garm-cli debug-log.
enable_log_streamer = false
# Enable the golang debug server. See the documentation in the "doc" folder for more information.
debug_server = false
```
## The callback_url option
Your runners will call back home with status updates as they install. Once they are set up, they will also send the GitHub agent ID they were allocated. You will need to configure the ```callback_url``` option in the ```garm``` server config. This URL needs to point to the following API endpoint:
```txt
POST /api/v1/callbacks/status
```
Example of a runner sending status updates:
```bash
garm-cli runner show garm-DvxiVAlfHeE7
+-----------------+------------------------------------------------------------------------------------+
| FIELD | VALUE |
+-----------------+------------------------------------------------------------------------------------+
| ID | 16b96ba2-d406-45b8-ab66-b70be6237b4e |
| Provider ID | garm-DvxiVAlfHeE7 |
| Name | garm-DvxiVAlfHeE7 |
| OS Type | linux |
| OS Architecture | amd64 |
| OS Name | ubuntu |
| OS Version | jammy |
| Status | running |
| Runner Status | idle |
| Pool ID | 8ec34c1f-b053-4a5d-80d6-40afdfb389f9 |
| Addresses | 10.198.117.120 |
| Status Updates | 2023-07-08T06:26:46: runner registration token was retrieved |
| | 2023-07-08T06:26:46: using cached runner found in /opt/cache/actions-runner/latest |
| | 2023-07-08T06:26:50: configuring runner |
| | 2023-07-08T06:26:56: runner successfully configured after 1 attempt(s) |
| | 2023-07-08T06:26:56: installing runner service |
| | 2023-07-08T06:26:56: starting service |
| | 2023-07-08T06:26:57: runner successfully installed |
+-----------------+------------------------------------------------------------------------------------+
```
This URL must be set and must be accessible by the instance. If you wish to restrict access to it, a reverse proxy can be configured to accept requests only from networks in which the runners ```garm``` manages will be spun up. This URL doesn't need to be globally accessible, it just needs to be accessible by the instances.
For example, in a scenario where you expose the API endpoint directly, this setting could look like the following:
```toml
callback_url = "https://garm.example.com/api/v1/callbacks/status"
```
Authentication is done using a short-lived JWT token, that gets generated for a particular instance that we are spinning up. That JWT token grants access to the instance to only update it's own status and to fetch metadata for itself. No other API endpoints will work with that JWT token. The validity of the token is equal to the pool bootstrap timeout value (default 20 minutes) plus the garm polling interval (5 minutes).
There is a sample ```nginx``` config [in the testdata folder](/testdata/nginx-server.conf). Feel free to customize it whichever way you see fit.
## The metadata_url option
The metadata URL is the base URL for any information an instance may need to fetch in order to finish setting itself up. As this URL may be placed behind a reverse proxy, you'll need to configure it in the ```garm``` config file. Ultimately this URL will need to point to the following ```garm``` API endpoint:
```bash
GET /api/v1/metadata
```
This URL needs to be accessible only by the instances ```garm``` sets up. This URL will not be used by anyone else. To configure it in ```garm``` add the following line in the ```[default]``` section of your ```garm``` config:
```toml
metadata_url = "https://garm.example.com/api/v1/metadata"
```
## The debug_server option
GARM can optionally enable the golang profiling server. This is useful if you suspect garm may be bottlenecking in any way. To enable the profiling server, add the following section to the garm config:
```toml
[default]
debug_server = true
```
And restart garm. You can then use the following command to start profiling:
```bash
go tool pprof http://127.0.0.1:9997/debug/pprof/profile?seconds=120
```
Important note on profiling when behind a reverse proxy. The above command will hang for a fairly long time. Most reverse proxies will timeout after about 60 seconds. To avoid this, you should only profile on localhost by connecting directly to garm.
It's also advisable to exclude the debug server URLs from your reverse proxy and only make them available locally.
Now that the debug server is enabled, here is a blog post on how to profile golang applications: https://blog.golang.org/profiling-go-programs
## The log_file option
By default, GARM logs everything to standard output.
You can optionally log to file by adding the following to your config file:
```toml
[default]
# Use this if you'd like to log to a file instead of standard output.
log_file = "/tmp/runner-manager.log"
```
### Rotating log files
GARM automatically rotates the log if it reaches 500 MB in size or 28 days, whichever comes first.
However, if you want to manually rotate the log file, you can send a `SIGHUP` signal to the GARM process.
You can add the following to your systemd unit file to enable `reload`:
```ini
[Service]
ExecReload=/bin/kill -HUP $MAINPID
```
Then you can simply:
```bash
systemctl reload garm
```
## The enable_log_streamer option
This option allows you to stream garm logs directly to your terminal. Set this option to true, then you can use the following command to stream logs:
```bash
garm-cli debug-log
```
An important note on enabling this option when behind a reverse proxy. The log streamer uses websockets to stream logs to you. You will need to configure your reverse proxy to allow websocket connections. If you're using nginx, you will need to add the following to your nginx `server` config:
```nginx
location /api/v1/ws {
proxy_pass http://garm_backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
}
```

18
doc/config_jwt_auth.md Normal file
View file

@ -0,0 +1,18 @@
# The JWT authentication config section
This section configures the JWT authentication used by the API server. GARM is currently a single user system and that user has the right to do anything and everything GARM is capable of. As a result, the JWT auth we have does not include a refresh token. The token is valid for the duration of the time to live (TTL) set in the config file. Once the token expires, you will need to log in again.
It is recommended that the secret be a long, randomly generated string. Changing the secret at any time will invalidate all existing tokens.
```toml
[jwt_auth]
# A JWT token secret used to sign tokens. Obviously, this needs to be changed :).
secret = ")9gk_4A6KrXz9D2u`0@MPea*sd6W`%@5MAWpWWJ3P3EqW~qB!!(Vd$FhNc*eU4vG"
# Time to live for tokens. Both the instances and you will use JWT tokens to
# authenticate against the API. However, this TTL is applied only to tokens you
# get when logging into the API. The tokens issued to the instances we manage,
# have a TTL based on the runner bootstrap timeout set on each pool. The minimum
# TTL for this token is 24h.
time_to_live = "8760h"
```

55
doc/config_metrics.md Normal file
View file

@ -0,0 +1,55 @@
# The metrics section
This is one of the features in GARM that I really love having. For one thing, it's community contributed and for another, it really adds value to the project. It allows us to create some pretty nice visualizations of what is happening with GARM.
At the moment there are only three meaningful metrics being collected, besides the default ones that the prometheus golang package enables by default. These are:
* `garm_health` - This is a gauge that is set to 1 if GARM is healthy and 0 if it is not. This is useful for alerting.
* `garm_runner_status` - This is a gauge value that gives us details about the runners garm spawns
* `garm_webhooks_received` - This is a counter that increments every time GARM receives a webhook from GitHub.
More metrics will be added in the future.
## Enabling metrics
Metrics are disabled by default. To enable them, add the following to your config file:
```toml
[metrics]
# Toggle metrics. If set to false, the API endpoint for metrics collection will
# be disabled.
enable = true
# Toggle to disable authentication (not recommended) on the metrics endpoint.
# If you do disable authentication, I encourage you to put a reverse proxy in front
# of garm and limit which systems can access that particular endpoint. Ideally, you
# would enable some kind of authentication using the reverse proxy, if the built-in auth
# is not sufficient for your needs.
disable_auth = false
```
You can choose to disable authentication if you wish, however it's not terribly difficult to set up, so I generally advise against disabling it.
## Configuring prometheus
The following section assumes that your garm instance is running at `garm.example.com` and has TLS enabled.
First, generate a new JWT token valid only for the metrics endpoint:
```bash
garm-cli metrics-token create
```
Note: The token validity is equal to the TTL you set in the [JWT config section](/doc/config_jwt_auth.md).
Copy the resulting token, and add it to your prometheus config file. The following is an example of how to add garm as a target in your prometheus config file:
```yaml
scrape_configs:
- job_name: "garm"
# Connect over https. If you don't have TLS enabled, change this to http.
scheme: https
static_configs:
- targets: ["garm.example.com"]
authorization:
credentials: "superSecretTokenYouGeneratedEarlier"
```

51
doc/database.md
View file

@ -1,37 +1,20 @@
# Database configuration
Garm currently supports two database backends:
GARM currently supports SQLite3. Support for other stores will be added in the future.
* SQLite3
* MySQL
You can choose either one of these. For most cases, ```SQLite3``` should do, but feel free to go with MySQL if you wish.
```toml
[database]
# Turn on/off debugging for database queries.
debug = false
# Database backend to use. Currently supported backends are:
# * sqlite3
# * mysql
backend = "sqlite3"
# the passphrase option is a temporary measure by which we encrypt the webhook
# secret that gets saved to the database, using AES256. In the future, secrets
# will be saved to something like Barbican or Vault, eliminating the need for
# this.
passphrase = "n<$n&P#L*TWqOh95_bN5J1r4mhxY7R84HZ%pvM#1vxJ<7~q%YVsCwU@Z60;7~Djo"
[database.mysql]
# If MySQL is used, these are the credentials and connection information used
# to connect to the server instance.
# database username
username = ""
# Database password
password = ""
# hostname to connect to
hostname = ""
# database name
database = ""
[database.sqlite3]
# Path on disk to the sqlite3 database file.
db_file = "/home/runner/file.db"
```
```toml
[database]
# Turn on/off debugging for database queries.
debug = false
# Database backend to use. Currently supported backends are:
# * sqlite3
backend = "sqlite3"
# the passphrase option is a temporary measure by which we encrypt the webhook
# secret that gets saved to the database, using AES256. In the future, secrets
# will be saved to something like Barbican or Vault, eliminating the need for
# this.
passphrase = "n<$n&P#L*TWqOh95_bN5J1r4mhxY7R84HZ%pvM#1vxJ<7~q%YVsCwU@Z60;7~Djo"
[database.sqlite3]
# Path on disk to the sqlite3 database file.
db_file = "/home/runner/garm.db"
```

BIN
doc/images/input_url.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 210 KiB

BIN
doc/images/jobs.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 62 KiB

BIN
doc/images/select_events.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 218 KiB

BIN
doc/images/tls_config.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 234 KiB

BIN
doc/images/webhooks.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 120 KiB

17
doc/logging.md
View file

@ -1,17 +0,0 @@
# Logging
By default, GARM is logging only on standard output.
If you would like GARM to use a logging file instead, you can use the `log_file` configuration option:
```toml
[default]
# Use this if you'd like to log to a file instead of standard output.
log_file = "/tmp/runner-manager.log"
```
## Rotating log files
If GARM uses a log file, by default it will rotate it when it reaches 500MB or 28 days, whichever comes first.
However, if you want to manually rotate the log file, you can send a `SIGHUP` signal to the GARM process.

2
doc/performance_considerations.md
View file

@ -2,7 +2,7 @@
Performance is often important when running GitHub action runners with garm. This document shows some ways to improve the creation time of a GitHub action runner.
## garm specific performance considerations
## GARM specific performance considerations
### Bundle the GitHub action runner

70
doc/providers.md
View file

@ -1,35 +1,38 @@
# Provider configuration
Garm was designed to be extensible. The database layer as well as the providers are defined as interfaces. Currently there are two providers:
GARM was designed to be extensible. Providers can be written either as built-in plugins or as external executables. The built-in plugins are written in Go, and they are compiled into the ```GARM``` binary. External providers are executables that implement the needed interface to create/delete/list compute systems that are used by ```GARM``` to create runners.
* [LXD](https://linuxcontainers.org/lxd/introduction/)
* External
GARM currently ships with one built-in provider for [LXD](https://linuxcontainers.org/lxd/introduction/) and the external provider interface which allows you to write your own provider in any language you want.
LXD is the simplest cloud-like system you can easily set up on any GNU/Linux machine, which enables you to create both containers and Virtual Machines. The ```external``` provider is a special type of provider, which delegates functionality to external executables.
- [LXD provider](#lxd-provider)
- [LXD remotes](#lxd-remotes)
- [LXD Security considerations](#lxd-security-considerations)
- [External provider](#external-provider)
- [Available external providers](#available-external-providers)
## The LXD provider
## LXD provider
Garm leverages the virtual machines feature of LXD to create the runners. Here is a sample config section for an LXD provider:
GARM leverages LXD to create the runners. Here is a sample config section for an LXD provider:
```toml
# Currently, providers are defined statically in the config. This is due to the fact
# that we have not yet added support for storing secrets in something like Barbican
# or Vault. This will change in the future. However, for now, it's important to remember
# that once you create a pool using one of the providers defined here, the name of that
# provider must not be changes, or the pool will no longer work. Make sure you remove any
# provider must not be changed, or the pool will no longer work. Make sure you remove any
# pools before removing or changing a provider.
[[provider]]
# An arbitrary string describing this provider.
name = "lxd_local"
# Provider type. Garm is designed to allow creating providers which are used to spin
# Provider type. GARM is designed to allow creating providers which are used to spin
# up compute resources, which in turn will run the github runner software.
# Currently, LXD is the only supported provider, but more will be written in the future.
# Currently, LXD is the only supprted provider, but more will be written in the future.
provider_type = "lxd"
# A short description of this provider. The name, description and provider types will
# be included in the information returned by the API when listing available providers.
description = "Local LXD installation"
[provider.lxd]
# the path to the unix socket that LXD is listening on. This works if garm and LXD
# the path to the unix socket that LXD is listening on. This works if GARM and LXD
# are on the same system, and this option takes precedence over the "url" option,
# which connects over the network.
unix_socket_path = "/var/snap/lxd/common/lxd/unix.socket"
@ -39,6 +42,14 @@ Garm leverages the virtual machines feature of LXD to create the runners. Here i
# This option allows you to inject the "default" profile along with the profile selected
# by the flavor.
include_default_profile = false
# instance_type defines the type of instances this provider will create.
#
# Options are:
#
# * virtual-machine (default)
# * container
#
instance_type = "container"
# enable/disable secure boot. If the image you select for the pool does not have a
# signed bootloader, set this to false, otherwise your instances won't boot.
secure_boot = false
@ -46,7 +57,7 @@ Garm leverages the virtual machines feature of LXD to create the runners. Here i
project_name = "default"
# URL is the address on which LXD listens for connections (ex: https://example.com:8443)
url = ""
# garm supports certificate authentication for LXD remote connections. The easiest way
# GARM supports certificate authentication for LXD remote connections. The easiest way
# to get the needed certificates, is to install the lxc client and add a remote. The
# client_certificate, client_key and tls_server_certificate can be then fetched from
# $HOME/snap/lxd/common/config.
@ -88,7 +99,7 @@ You can choose to connect to a local LXD server by using the ```unix_socket_path
### LXD remotes
By default, garm does not load any image remotes. You get to choose which remotes you add (if any). An image remote is a repository of images that LXD uses to create new instances, either virtual machines or containers. In the absence of any remote, garm will attempt to find the image you configure for a pool of runners, on the LXD server we're connecting to. If one is present, it will be used, otherwise it will fail and you will need to configure a remote.
By default, GARM does not load any image remotes. You get to choose which remotes you add (if any). An image remote is a repository of images that LXD uses to create new instances, either virtual machines or containers. In the absence of any remote, GARM will attempt to find the image you configure for a pool of runners, on the LXD server we're connecting to. If one is present, it will be used, otherwise it will fail and you will need to configure a remote.
The sample config file in this repository has the usual default ```LXD``` remotes:
@ -100,20 +111,28 @@ When creating a new pool, you'll be able to specify which image you want to use.
You can also create your own image remote, where you can host your own custom images. If you want to build your own images, have a look at [distrobuilder](https://github.com/lxc/distrobuilder).
Image remotes in the ```garm``` config, is a map of strings to remote settings. The name of the remote is the last bit of string in the section header. For example, the following section ```[provider.lxd.image_remotes.ubuntu_daily]```, defines the image remote named **ubuntu_daily**. Use this name to reference images inside that remote.
Image remotes in the ```GARM``` config, is a map of strings to remote settings. The name of the remote is the last bit of string in the section header. For example, the following section ```[provider.lxd.image_remotes.ubuntu_daily]```, defines the image remote named **ubuntu_daily**. Use this name to reference images inside that remote.
## The External provider
You can also use locally uploaded images. Check out the [performance considerations](./performance_considerations.md) page for details on how to customize local images and use them with GARM.
### LXD Security considerations
GARM does not apply any ACLs of any kind to the instances it creates. That task remains in the responsibility of the user. [Here is a guide for creating ACLs in LXD](https://linuxcontainers.org/lxd/docs/master/howto/network_acls/). You can of course use ```iptables``` or ```nftables``` to create any rules you wish. I recommend you create a separate isolated lxd bridge for runners, and secure it using ACLs/iptables/nftables.
You must make sure that the code that runs as part of the workflows is trusted, and if that cannot be done, you must make sure that any malicious code that will be pulled in by the actions and run as part of a workload, is as contained as possible. There is a nice article about [securing your workflow runs here](https://blog.gitguardian.com/github-actions-security-cheat-sheet/).
## External provider
The external provider is a special kind of provider. It delegates the functionality needed to create the runners to external executables. These executables can be either binaries or scripts. As long as they adhere to the needed interface, they can be used to create runners in any target IaaS. This is identical to what ```containerd``` does with ```CNIs```.
There is currently one external provider for [OpenStack](https://www.openstack.org/) available in the [contrib folder of this repository](../contrib/providers.d/openstack). The provider is written in ```bash``` and it is just a sample. A production ready provider would need more error checking and idempotency, but it serves as an example of what can be done. As it stands, it is functional.
There are currently two sample external providers available in the [contrib folder of this repository](../contrib/providers.d/). The providers are written in ```bash``` and are meant as examples of how a provider could be written in ```bash```. Production ready providers would need more error checking and idempotency, but they serve as an example of what can be done. As it stands, they are functional.
The configuration for an external provider is quite simple:
```toml
# This is an example external provider. External providers are executables that
# implement the needed interface to create/delete/list compute systems that are used
# by garm to create runners.
# by GARM to create runners.
[[provider]]
name = "openstack_external"
description = "external openstack provider"
@ -127,13 +146,24 @@ provider_type = "external"
provider_executable = "/etc/garm/providers.d/openstack/garm-external-provider"
```
The external provider has three options:
The external provider has two options:
* ```provider_executable```
* ```config_file```
The ```provider_executable``` option is the absolute path to an executable that implements the provider logic. Garm will delegate all provider operations to this executable. This executable can be anything (bash, python, perl, go, etc). See [Writing an external provider](./external_provider.md) for more details.
The ```provider_executable``` option is the absolute path to an executable that implements the provider logic. GARM will delegate all provider operations to this executable. This executable can be anything (bash, python, perl, go, etc). See [Writing an external provider](./external_provider.md) for more details.
The ```config_file``` option is a path on disk to an arbitrary file, that is passed to the external executable via the environment variable ```GARM_PROVIDER_CONFIG_FILE```. This file is only relevant to the external provider. Garm itself does not read it. In the case of the OpenStack provider, this file contains access information for an OpenStack cloud (what you would typically find in a ```keystonerc``` file) as well as some provider specific options like whether or not to boot from volume and which tenant network to use. You can check out the [sample config file](../contrib/providers.d/openstack/keystonerc) in this repository.
The ```config_file``` option is a path on disk to an arbitrary file, that is passed to the external executable via the environment variable ```GARM_PROVIDER_CONFIG_FILE```. This file is only relevant to the external provider. GARM itself does not read it. In the case of the sample OpenStack provider, this file contains access information for an OpenStack cloud (what you would typically find in a ```keystonerc``` file) as well as some provider specific options like whether or not to boot from volume and which tenant network to use. You can check out the [sample config file](../contrib/providers.d/openstack/keystonerc) in this repository.
If you want to implement an external provider, you can use this file for anything you need to pass into the binary when ```garm``` calls it to execute a particular operation.
If you want to implement an external provider, you can use this file for anything you need to pass into the binary when ```GARM``` calls it to execute a particular operation.
### Available external providers
For non testing purposes, there are two external providers currently available:
* [OpenStack](https://github.com/cloudbase/garm-provider-openstack)
* [Azure](https://github.com/cloudbase/garm-provider-azure)
Details on how to install and configure them are available in their respective repositories.
If you wrote a provider and would like to add it to the above list, feel free to open a PR.

551
doc/quickstart.md Normal file
View file

@ -0,0 +1,551 @@
# Quick start
Okay, I lied. It's not that quick. But it's not that long either. I promise.
In this guide I'm going to take you through the entire process of setting up garm from scratch. This will include editing the config file (which will probably take the longest amount of time), fetching a proper [PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) (personal access token) from GitHub, setting up the webhooks endpoint, defining your repo/org/enterprise and finally setting up a runner pool.
For the sake of this guide, we'll assume you have access to the following setup:
* A linux machine (ARM64 or AMD64)
* Optionally, docker/podman installed on that machine
* A public IP address or port forwarding set up on your router for port `80` or `443`. You can forward any ports, we just need to remember to use the same ports when we define the webhook in github, and the two URLs in the config file (more on that later). For the sake of this guide, I will assume you have port `80` or `443` forwarded to your machine.
* An `A` record pointing to your public IP address (optional, but recommended). Alternatively, you can use the IP address directly. I will use `garm.example.com` in this guide. If you'll be using an IP address, just replace `garm.example.com` with your IP address throughout this guide.
* All config files and data will be stored in `/etc/garm`.
* A [Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
Why the need to expose GARM to the internet? Well, GARM uses webhooks sent by GitHub to automatically scale runners. Whenever a new job starts, a webhook is generated letting GARM know that there is a need for a runner. GARM then spins up a new runner instance and registers it with GitHub. When the job is done, the runner instance is automatically removed. This workflow is enabled by webhooks.
## The GitHub PAT (Personal Access Token)
Let's start by fetching a PAT so we get that out of the way. You can use the [GitHub docs](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) to create a PAT.
For a `classic` PAT, GARM needs the following permissions to function properly (depending on the hierarchy level you want to manage):
* ```public_repo``` - for access to a repository
* ```repo``` - for access to a private repository
* ```admin:org``` - if you plan on using this with an organization to which you have access
* ```manage_runners:enterprise``` - if you plan to use garm at the enterprise level
This doc will be updated at a future date with the exact permissions needed in case you want to use a fine grained PAT.
## Create the config folder
All of our config files and data will be stored in `/etc/garm`. Let's create that folder:
```bash
sudo mkdir -p /etc/garm
```
Coincidentally, this is also where the docker container [looks for the config](../Dockerfile#L29) when it starts up. You can either use `Docker` or you can set up garm directly on your system. I'll show you both ways. In both cases, we need to first create the config folder and a proper config file.
## The config file
There is a full config file, with detailed comments for each option, in the [testdata folder](../testdata/config.toml). You can use that as a reference. But for the purposes of this guide, we'll be using a minimal config file and add things on as we proceed.
Open `/etc/garm/config.toml` in your favorite editor and paste the following:
```toml
[default]
callback_url = "https://garm.example.com/api/v1/callbacks/status"
metadata_url = "https://garm.example.com/api/v1/metadata"
[metrics]
enable = true
disable_auth = false
[jwt_auth]
# Obviously, this needs to be changed :).
secret = ")9gk_4A6KrXz9D2u`0@MPea*sd6W`%@5MAWpWWJ3P3EqW~qB!!(Vd$FhNc*eU4vG"
time_to_live = "8760h"
[apiserver]
bind = "0.0.0.0"
port = 80
use_tls = false
[database]
backend = "sqlite3"
# This needs to be changed.
passphrase = "shreotsinWadquidAitNefayctowUrph"
[database.sqlite3]
db_file = "/etc/garm/garm.db"
```
This is a minimal config, with no providers or credentials defined. In this example we have the [default](./config_default.md), [metrics](./config_metrics.md), [jwt_auth](./config_jwt_auth.md), [apiserver](./config_api_server.md) and [database](./config_database.md) sections. Each are documented separately. Feel free to read through the available docs if, for example you need to enable TLS without using an nginx reverse proxy or if you want to enable the debug server, the log streamer or a log file.
In this sample config we:
* define the callback and the metadata URLs
* enable metrics with authentication
* set a JWT secret which is used to sign JWT tokens
* set a time to live for the JWT tokens
* enable the API server on port `80` and bind it to all interfaces
* set the database backend to `sqlite3` and set a passphrase for sealing secrets (just webhook secrets for now)
The callback URLs are really important and need to point back to garm. You will notice that the domain name used in these options, is the same one we defined at the beginning of this guide. If you won't use a domain name, replace `garm.example.com` with your IP address and port number.
We need to tell garm by which addresses it can be reached. There are many ways by which GARMs API endpoints can be exposed, and there is no sane way in which GARM itself can determine if it's behind a reverse proxy or not. The metadata URL may be served by a reverse proxy with a completely different domain name than the callback URL. Both domains pointing to the same installation of GARM in the end.
The information in these two options is used by the instances we spin up to phone home their status and to fetch the needed metadata to finish setting themselves up. For now, the metadata URL is only used to fetch the runner registration token.
We won't go too much into detail about each of the options here. Have a look at the different config sections and their respective docs for more information.
At this point, we have a valid config file, but we still need to add `provider` and `credentials` sections.
## The provider section
This is where you have a decision to make. GARM has a number of providers you can leverage. At the time of this writing, we have support for:
* LXD
* Azure
* OpenStack
The LXD provider is built into GARM itself and has no external requirements. The [Azure](https://github.com/cloudbase/garm-provider-azure) and [OpenStack](https://github.com/cloudbase/garm-provider-openstack) ones are `external` providers in the form of an executable that GARM calls into.
Both the LXD and the external provider configs are [documented in a separate doc](./providers.md).
The easiest provider to set up is probably the LXD provider. You don't need an account on an external cloud. You can just use your machine.
You will need to have LXD installed and configured. There is an excellent [getting started guide](https://documentation.ubuntu.com/lxd/en/latest/getting_started/) for LXD. Follow the instructions there to install and configure LXD, then come back here.
Once you have LXD installed and configured, you can add the provider section to your config file. If you're connecting to the `local` LXD installation, the [config snippet for the LXD provider](./providers.md#lxd-provider) will work out of the box. We'll be connecting using the unix socket so no further configuration will be needed.
Go ahead and copy and paste that entire snippet in your GARM config file (`/etc/garm/config.toml`).
You can also use an external provider instead of LXD. You will need to define the provider section in your config file and point it to the executable and the provider config file. The [config snippet for the external provider](./providers.md#external-provider) gives you an example of how that can be done. Configuring the external provider is outside the scope of this guide. You will need to consult the documentation for the external provider you want to use.
## The credentials section
The credentials section is where we define out GitHub credentials. GARM is capable of using either GitHub proper or [GitHub Enterprise Server](https://docs.github.com/en/enterprise-server@3.6/get-started/onboarding/getting-started-with-github-enterprise-server). The credentials section allows you to override the default GitHub API endpoint and point it to your own deployment of GHES.
The credentials section is [documented in a separate doc](./github_credentials.md), but we will include a small snippet here for clarity.
```toml
# This is a list of credentials that you can define as part of the repository
# or organization definitions. They are not saved inside the database, as there
# is no Vault integration (yet). This will change in the future.
# Credentials defined here can be listed using the API. Obviously, only the name
# and descriptions are returned.
[[github]]
name = "gabriel"
description = "github token for user gabriel"
# This is a personal token with access to the repositories and organizations
# you plan on adding to garm. The "workflow" option needs to be selected in order
# to work with repositories, and the admin:org needs to be set if you plan on
# adding an organization.
oauth2_token = "super secret token"
```
The `oauth2_token` option will hold the PAT we created earlier. You can add multiple credentials to the config file. Each will be referenced by name when we define the repo/org/enterprise.
Alright, we're almost there. We have a config file with a provider and a credentials section. We now have to start the service and create a webhook in GitHub pointing at our `webhook` endpoint.
## Starting the service
You can start GARM using docker or directly on your system. I'll show you both ways.
### Using Docker
If you're using docker, you can start the service with:
```bash
docker run -d \
--name garm \
-p 80:80 \
-v /etc/garm:/etc/garm:rw \
-v /var/snap/lxd/common/lxd/unix.socket:/var/snap/lxd/common/lxd/unix.socket:rw \
ghcr.io/cloudbase/garm:v0.1.2
```
You will notice we also mounted the LXD unix socket from the host inside the container where the config you pasted expects to find it. If you plan to use an external provider that does not need to connect to LXD over a unix socket, feel free to remove that mount.
Check the logs to make sure everything is working as expected:
```bash
ubuntu@garm:~$ docker logs garm
signal.NotifyContext(context.Background, [interrupt terminated])
2023/07/17 21:55:43 Loading provider lxd_local
2023/07/17 21:55:43 registering prometheus metrics collectors
2023/07/17 21:55:43 setting up metric routes
```
### Setting up GARM as a system service
This process is a bit more involved. We'll need to create a new user for garm and set up permissions for that user to connect to LXD.
First, create the user:
```bash
useradd --shell /usr/bin/false \
--system \
--groups lxd \
--no-create-home garm
```
Adding the `garm` user to the LXD group will allow it to connect to the LXD unix socket. We'll need that considering the config we crafted above.
Next, download the latest release from the [releases page](https://github.com/cloudbase/garm/releases).
```bash
wget -q -O - https://github.com/cloudbase/garm/releases/download/v0.1.2/garm-linux-amd64.tgz | tar xzf - -C /usr/local/bin/
```
We'll be running under an unprivileged user. If we want to be able to listen on any port under `1024`, we'll have to set some capabilities on the binary:
```bash
setcap cap_net_bind_service=+ep /usr/local/bin/garm
```
Change the permissions on the config dir:
```bash
chown -R garm:garm /etc/garm
```
Copy the sample `systemd` service file:
```bash
wget -O /etc/systemd/system/garm.service \
https://raw.githubusercontent.com/cloudbase/garm/v0.1.2/contrib/garm.service
```
Reload the `systemd` daemon and start the service:
```bash
systemctl daemon-reload
systemctl start garm
```
Check the logs to make sure everything is working as expected:
```bash
ubuntu@garm:~$ sudo journalctl -u garm
```
Check that you can make a request to the API:
```bash
ubuntu@garm:~$ curl http://garm.example.com/webhooks
ubuntu@garm:~$ docker logs garm
signal.NotifyContext(context.Background, [interrupt terminated])
2023/07/17 22:21:33 Loading provider lxd_local
2023/07/17 22:21:33 registering prometheus metrics collectors
2023/07/17 22:21:33 setting up metric routes
2023/07/17 22:21:35 ignoring unknown event
172.17.0.1 - - [17/Jul/2023:22:21:35 +0000] "GET /webhooks HTTP/1.1" 200 0 "" "curl/7.81.0"
```
Excellent! We have a working GARM installation. Now we need to set up the webhook in GitHub.
## Setting up the webhook
Before we create a pool, we need to set up the webhook in GitHub. This is a fairly simple process.
Head over to the [webhooks doc](./webhooks.md) and follow the instructions there. Come back here when you're done.
After you've finished setting up the webhook, there are just a few more things to do:
* Initialize GARM
* Add a repo/org/enterprise
* Create a pool
## Initializing GARM
Before we can start using GARM, we need initialize it. This will create the `admin` user and generate a unique controller ID that will identify this GARM installation. This process allows us to use multiple GARM installations with the same GitHub account. GARM will use the controller ID to identify the runners it creates. This way we won't run the risk of accidentally removing runners we don't manage.
To initialize GARM, we'll use the `garm-cli` tool. You can download the latest release from the [releases page](https://github.com/cloudbase/garm/releases):
```bash
wget -q -O - https://github.com/cloudbase/garm/releases/download/v0.1.2/garm-cli-linux-amd64.tgz | tar xzf - -C /usr/local/bin/
```
Now we can initialize GARM:
```bash
ubuntu@garm:~$ garm-cli init --name="local_garm" --url https://garm.example.com
Username: admin
Email: root@localhost
✔ Password: *************
+----------+--------------------------------------+
| FIELD | VALUE |
+----------+--------------------------------------+
| ID | ef4ab6fd-1252-4d5a-ba5a-8e8bd01610ae |
| Username | admin |
| Email | root@localhost |
| Enabled | true |
+----------+--------------------------------------+
```
The init command also created a local CLI profile for your new GARM server:
```bash
ubuntu@garm:~# garm-cli profile list
+----------------------+--------------------------+
| NAME | BASE URL |
+----------------------+--------------------------+
| local_garm (current) | https://garm.example.com |
+----------------------+--------------------------+
```
Every time you init a new GARM instance, a new profile will be created in your local `garm-cli` config. You can also log into an already initialized instance using:
```bash
garm-cli profile add --name="another_garm" --url https://garm2.example.com
```
Then you can switch between profiles using:
```bash
garm-cli profile switch another_garm
```
## Define a repo
We now have a working GARM installation, with github credentials and a provider added. It's time to add a repo.
Before we add a repo, let's list credentials. We'll need their names when we'll add a new repo.
```bash
gabriel@rossak:~$ garm-cli credentials list
+---------+-------------------------------+--------------------+-------------------------+-----------------------------+
| NAME | DESCRIPTION | BASE URL | API URL | UPLOAD URL |
+---------+-------------------------------+--------------------+-------------------------+-----------------------------+
| gabriel | github token for user gabriel | https://github.com | https://api.github.com/ | https://uploads.github.com/ |
+---------+-------------------------------+--------------------+-------------------------+-----------------------------+
```
Even though you didn't explicitly set the URLs, GARM will default to the GitHub ones. You can override them if you want to use a GHES deployment.
Now we can add a repo:
```bash
garm-cli repo add \
--credentials gabriel \
--owner gsamfira \
--name scripts \
--webhook-secret $SECRET
```
In this case, `$SECRET` holds the webhook secret you set previously when you defined the webhook in GitHub. This secret is mandatory as GARM will always validate the webhook payloads it receives.
You should see something like this:
```bash
gabriel@rossak:~$ garm-cli repo add \
> --credentials gabriel \
> --owner gsamfira \
> --name scripts \
> --webhook-secret $SECRET
+----------------------+--------------------------------------+
| FIELD | VALUE |
+----------------------+--------------------------------------+
| ID | f4900c7c-2ec0-41bd-9eab-d70fe9bd850d |
| Owner | gsamfira |
| Name | scripts |
| Credentials | gabriel |
| Pool manager running | false |
| Failure reason | |
+----------------------+--------------------------------------+
```
We can now list the repos:
```bash
gabriel@rock:~$ garm-cli repo ls
+--------------------------------------+----------+---------+------------------+------------------+
| ID | OWNER | NAME | CREDENTIALS NAME | POOL MGR RUNNING |
+--------------------------------------+----------+---------+------------------+------------------+
| f4900c7c-2ec0-41bd-9eab-d70fe9bd850d | gsamfira | scripts | gabriel | true |
+--------------------------------------+----------+---------+------------------+------------------+
```
Excellent! Make a note of the ID. We'll need it later when we create a pool.
## Create a pool
This is the last step. You're almost there!
To create a pool we'll need the repo ID from the previous step (which we have) and a provider in which the pool will spin up new runners. We'll use the LXD provider we defined earlier, but we need its name:
```bash
gabriel@rossak:~$ garm-cli provider list
+-----------+------------------------+------+
| NAME | DESCRIPTION | TYPE |
+-----------+------------------------+------+
| lxd_local | Local LXD installation | lxd |
+-----------+------------------------+------+
```
Now we can create a pool:
```bash
garm-cli pool add \
--repo f4900c7c-2ec0-41bd-9eab-d70fe9bd850d \
--enabled true \
--provider-name lxd_local \
--flavor default \
--image ubuntu:22.04 \
--max-runners 5 \
--min-idle-runners 0 \
--os-arch amd64 \
--os-type linux \
--tags ubuntu,generic
```
You should see something like this:
```bash
gabriel@rossak:~$ garm-cli pool add \
> --repo f4900c7c-2ec0-41bd-9eab-d70fe9bd850d \
> --enabled true \
> --provider-name lxd_local \
> --flavor default \
> --image ubuntu:22.04 \
> --max-runners 5 \
> --min-idle-runners 0 \
> --os-arch amd64 \
> --os-type linux \
> --tags ubuntu,generic
+--------------------------+--------------------------------------------+
| FIELD | VALUE |
+--------------------------+--------------------------------------------+
| ID | 344e4a72-2035-4a18-a3d5-87bd3874b56c |
| Provider Name | lxd_local |
| Image | ubuntu:22.04 |
| Flavor | default |
| OS Type | linux |
| OS Architecture | amd64 |
| Max Runners | 5 |
| Min Idle Runners | 0 |
| Runner Bootstrap Timeout | 20 |
| Tags | self-hosted, amd64, Linux, ubuntu, generic |
| Belongs to | gsamfira/scripts |
| Level | repo |
| Enabled | true |
| Runner Prefix | garm |
| Extra specs | |
| GitHub Runner Group | |
+--------------------------+--------------------------------------------+
```
If we list the pool we should see it:
```bash
gabriel@rock:~$ garm-cli pool ls -a
+--------------------------------------+--------------+---------+----------------------------------------+------------------+-------+---------+---------------+
| ID | IMAGE | FLAVOR | TAGS | BELONGS TO | LEVEL | ENABLED | RUNNER PREFIX |
+--------------------------------------+--------------+---------+----------------------------------------+------------------+-------+---------+---------------+
| 344e4a72-2035-4a18-a3d5-87bd3874b56c | ubuntu:22.04 | default | self-hosted amd64 Linux ubuntu generic | gsamfira/scripts | repo | true | garm |
+--------------------------------------+--------------+---------+----------------------------------------+------------------+-------+---------+---------------+
```
This pool is enabled, but the `min-idle-runners` option is set to 0. This means that it will not create any lingering runners. It will only create runners when a job is started. If your provider is slow to boot up new instances, you may want to set this to a value higher than 0.
For the purposes of this guide, we'll increase it to 1 so we have a runner created.
First, list current runners:
```bash
gabriel@rossak:~$ garm-cli runner ls -a
+----+------+--------+---------------+---------+
| NR | NAME | STATUS | RUNNER STATUS | POOL ID |
+----+------+--------+---------------+---------+
+----+------+--------+---------------+---------+
```
No runners. Now, let's update the pool and set `min-idle-runners` to 1:
```bash
gabriel@rossak:~$ garm-cli pool update 344e4a72-2035-4a18-a3d5-87bd3874b56c --min-idle-runners=1
+--------------------------+--------------------------------------------+
| FIELD | VALUE |
+--------------------------+--------------------------------------------+
| ID | 344e4a72-2035-4a18-a3d5-87bd3874b56c |
| Provider Name | lxd_local |
| Image | ubuntu:22.04 |
| Flavor | default |
| OS Type | linux |
| OS Architecture | amd64 |
| Max Runners | 5 |
| Min Idle Runners | 1 |
| Runner Bootstrap Timeout | 20 |
| Tags | self-hosted, amd64, Linux, ubuntu, generic |
| Belongs to | gsamfira/scripts |
| Level | repo |
| Enabled | true |
| Runner Prefix | garm |
| Extra specs | |
| GitHub Runner Group | |
+--------------------------+--------------------------------------------+
```
Now if we list the runners:
```bash
gabriel@rossak:~$ garm-cli runner ls -a
+----+-------------------+----------------+---------------+--------------------------------------+
| NR | NAME | STATUS | RUNNER STATUS | POOL ID |
+----+-------------------+----------------+---------------+--------------------------------------+
| 1 | garm-tdtD6zpsXhj1 | pending_create | pending | 344e4a72-2035-4a18-a3d5-87bd3874b56c |
+----+-------------------+----------------+---------------+--------------------------------------+
```
If we check our LXD, we should also see it there as well:
```bash
gabriel@rossak:~$ lxc list
+-------------------+---------+---------------------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-------------------+---------+---------------------+------+-----------+-----------+
| garm-tdtD6zpsXhj1 | RUNNING | 10.44.30.155 (eth0) | | CONTAINER | 0 |
+-------------------+---------+---------------------+------+-----------+-----------+
```
If we wait for a bit and run:
```bash
gabriel@rossak:~$ garm-cli runner show garm-tdtD6zpsXhj1
+-----------------+------------------------------------------------------------------------------------------------------+
| FIELD | VALUE |
+-----------------+------------------------------------------------------------------------------------------------------+
| ID | 7ac024c9-1854-4911-9859-d061059244a6 |
| Provider ID | garm-tdtD6zpsXhj1 |
| Name | garm-tdtD6zpsXhj1 |
| OS Type | linux |
| OS Architecture | amd64 |
| OS Name | ubuntu |
| OS Version | jammy |
| Status | running |
| Runner Status | idle |
| Pool ID | 344e4a72-2035-4a18-a3d5-87bd3874b56c |
| Addresses | 10.44.30.155 |
| Status Updates | 2023-07-18T14:32:26: runner registration token was retrieved |
| | 2023-07-18T14:32:26: downloading tools from https://github.com/actions/runner/releases/download/v2.3 |
| | 06.0/actions-runner-linux-amd64-2.306.0.tar.gz |
| | 2023-07-18T14:32:30: extracting runner |
| | 2023-07-18T14:32:36: installing dependencies |
| | 2023-07-18T14:33:03: configuring runner |
| | 2023-07-18T14:33:14: runner successfully configured after 1 attempt(s) |
| | 2023-07-18T14:33:14: installing runner service |
| | 2023-07-18T14:33:15: starting service |
| | 2023-07-18T14:33:16: runner successfully installed |
+-----------------+------------------------------------------------------------------------------------------------------+
```
We can see the runner getting installed and phoning home with status updates. You should now see it in your GitHub repo under `Settings --> Actions --> Runners`.
You can also target this runner using one or more of its labels. In this case, we can target it using `ubuntu` or `generic`.
You can also view jobs sent to your garm instance using the `garm-cli job ls` command:
```bash
gabriel@rossak:~$ garm-cli job ls
+----+------+--------+------------+-------------+------------+------------------+-----------+
| ID | NAME | STATUS | CONCLUSION | RUNNER NAME | REPOSITORY | REQUESTED LABELS | LOCKED BY |
+----+------+--------+------------+-------------+------------+------------------+-----------+
+----+------+--------+------------+-------------+------------+------------------+-----------+
```
There are no jobs sent yet to my GARM install, but once you start sending jobs, you'll see them here as well.
That's it! You now have a working GARM installation. You can add more repos, orgs or enterprises and create more pools. You can also add more providers for different clouds and credentials with access to different GitHub resources.

391
doc/running_garm.md
View file

@ -1,391 +0,0 @@
# Running garm
Create a folder for the config:
```bash
mkdir $HOME/garm
```
Create a config file for ```garm```:
```bash
cp ./testdata/config.toml $HOME/garm/config.toml
```
Customize the config whichever way you want, then run ```garm```:
```bash
garm -config $HOME/garm/config.toml
```
This will start the API and migrate the database. Note, if you're using MySQL, you will need to create a database, grant access to a user and configure those credentials in the ```config.toml``` file.
## First run
Before you can use ```garm```, you need to initialize it. This means we need to create an admin user, and login:
```bash
ubuntu@experiments:~$ garm-cli init --name="local_garm" --url https://garm.example.com
Username: admin
Email: root@localhost
✔ Password: *************█
+----------+--------------------------------------+
| FIELD | VALUE |
+----------+--------------------------------------+
| ID | ef4ab6fd-1252-4d5a-ba5a-8e8bd01610ae |
| Username | admin |
| Email | root@localhost |
| Enabled | true |
+----------+--------------------------------------+
```
Alternatively you can run this in non-interactive mode. See ```garm-cli init -h``` for details.
## Enabling bash completion
Before we begin, let's make our lives a little easier and set up bash completion. The wonderful [cobra](https://github.com/spf13/cobra) library gives us completion for free:
```bash
mkdir $HOME/.bash_completion.d
echo 'source $HOME/.bash_completion.d/* >/dev/null 2>&1|| true' >> $HOME/.bash_completion
```
Now generate the completion file:
```bash
garm-cli completion bash > $HOME/.bash_completion.d/garm
```
Completion for multiple shells is available:
```bash
ubuntu@experiments:~$ garm-cli completion
Generate the autocompletion script for garm-cli for the specified shell.
See each sub-command's help for details on how to use the generated script.
Usage:
garm-cli completion [command]
Available Commands:
bash Generate the autocompletion script for bash
fish Generate the autocompletion script for fish
powershell Generate the autocompletion script for powershell
zsh Generate the autocompletion script for zsh
Flags:
-h, --help help for completion
Global Flags:
--debug Enable debug on all API calls
Use "garm-cli completion [command] --help" for more information about a command.
```
## Adding a repository/organization/enterprise
To add a repository, we need credentials. Let's list the available credentials currently configured. These credentials are added to ```garm``` using the config file (see above), but we need to reference them by name when creating a repo.
```bash
ubuntu@experiments:~$ garm-cli credentials list
+---------+------------------------------+
| NAME | DESCRIPTION |
+---------+------------------------------+
| gabriel | github token or user gabriel |
+---------+------------------------------+
```
Now we can add a repository to ```garm```:
```bash
ubuntu@experiments:~$ garm-cli repository create \
--credentials=gabriel \
--owner=gabriel-samfira \
--name=scripts \
--webhook-secret="super secret webhook secret you configured in github webhooks"
+-------------+--------------------------------------+
| FIELD | VALUE |
+-------------+--------------------------------------+
| ID | 77258e1b-81d2-4821-bdd7-f6923a026455 |
| Owner | gabriel-samfira |
| Name | scripts |
| Credentials | gabriel |
+-------------+--------------------------------------+
```
To add an organization, use the following command:
```bash
ubuntu@experiments:~$ garm-cli organization create \
--credentials=gabriel \
--name=gsamfira \
--webhook-secret="$SECRET"
+-------------+--------------------------------------+
| FIELD | VALUE |
+-------------+--------------------------------------+
| ID | 7f0b83d5-3dc0-42de-b189-f9bbf1ae8901 |
| Name | gsamfira |
| Credentials | gabriel |
+-------------+--------------------------------------+
```
To add an enterprise, use the following command:
```bash
ubuntu@experiments:~$ garm-cli enterprise create \
--credentials=gabriel \
--name=gsamfira \
--webhook-secret="$SECRET"
+-------------+--------------------------------------+
| FIELD | VALUE |
+-------------+--------------------------------------+
| ID | 0925033b-049f-4334-a460-c26f979d2356 |
| Name | gsamfira |
| Credentials | gabriel |
+-------------+--------------------------------------+
```
## Creating a pool
Pools are objects that define one type of worker and rules by which that pool of workers will be maintained. You can have multiple pools of different types of instances. Each pool can have different images, be on different providers and have different tags.
Before we can create a pool, we need to list the available providers. Providers are defined in the config (see above), but we need to reference them by name in the pool.
```bash
ubuntu@experiments:~$ garm-cli provider list
+-----------+------------------------+------+
| NAME | DESCRIPTION | TYPE |
+-----------+------------------------+------+
| lxd_local | Local LXD installation | lxd |
+-----------+------------------------+------+
```
Now we can create a pool for repo ```gabriel-samfira/scripts```:
```bash
ubuntu@experiments:~$ garm-cli pool add \
--repo=77258e1b-81d2-4821-bdd7-f6923a026455 \
--flavor="default" \
--image="ubuntu:20.04" \
--provider-name="lxd_local" \
--tags="ubuntu,simple-runner,repo-runner" \
--enabled=false
+------------------+-------------------------------------------------------------+
| FIELD | VALUE |
+------------------+-------------------------------------------------------------+
| ID | fb25f308-7ad2-4769-988e-6ec2935f642a |
| Provider Name | lxd_local |
| Image | ubuntu:20.04 |
| Flavor | default |
| OS Type | linux |
| OS Architecture | amd64 |
| Max Runners | 5 |
| Min Idle Runners | 1 |
| Tags | ubuntu, simple-runner, repo-runner, self-hosted, x64, linux |
| Belongs to | gabriel-samfira/scripts |
| Level | repo |
| Enabled | false |
+------------------+-------------------------------------------------------------+
```
There are a bunch of things going on here, so let's break it down. We created a pool for repo ```gabriel-samfira/scripts``` (identified by the ID ```77258e1b-81d2-4821-bdd7-f6923a026455```). This pool has the following characteristics:
* flavor=default - The **flavor** describes the hardware aspects of an instance. In LXD terms, this translates to [profiles](https://linuxcontainers.org/lxd/docs/master/profiles/). In LXD, profiles describe how much memory, CPU, NICs and disks a particular instance will get. Much like the flavors in OpenStack or any public cloud provider
* image=ubuntu:20.04 - The image describes the operating system that will be spun up on the provider. LXD fetches these images from one of the configured remotes, or from the locally cached images. On AWS, this would be an AMI (for example).
* provider-name=lxd_local - This is the provider on which we'll be spinning up runners. You can have as many providers defined as you wish, and you can reference either one of them when creating a pool.
* tags="ubuntu,simple-runner,repo-runner" - This list of tags will be added to all runners maintained by this pool. These are the tags you can use to target these runners in your workflows. By default, the github runner will automatically add a few default tags (self-hosted, x64, linux in the above example)
* enabled=false - This option creates the pool in **disabled** state. When disabled, no new runners will be spun up.
By default, a pool is created with a max worker count of ```5``` and a minimum idle runner count of ```1```. This means that this pool will create by default one runner, and will automatically add more, as jobs are triggered on github. The idea is to have at least one runner ready to accept a workflow job. The pool will keep adding workers until the max runner count is reached. Once a workflow job is complete, the runner is automatically deleted, and replaced.
To update the pool, we cam use the following command:
```bash
ubuntu@experiments:~$ garm-cli pool update fb25f308-7ad2-4769-988e-6ec2935f642a --enabled=true
+------------------+-------------------------------------------------------------+
| FIELD | VALUE |
+------------------+-------------------------------------------------------------+
| ID | fb25f308-7ad2-4769-988e-6ec2935f642a |
| Provider Name | lxd_local |
| Image | ubuntu:20.04 |
| Flavor | default |
| OS Type | linux |
| OS Architecture | amd64 |
| Max Runners | 5 |
| Min Idle Runners | 1 |
| Tags | ubuntu, simple-runner, repo-runner, self-hosted, x64, linux |
| Belongs to | gabriel-samfira/scripts |
| Level | repo |
| Enabled | true |
+------------------+-------------------------------------------------------------+
```
Now, if we list the runners, we should see one being created:
```bash
ubuntu@experiments:~$ garm-cli runner ls fb25f308-7ad2-4769-988e-6ec2935f642a
+-------------------------------------------+----------------+---------------+--------------------------------------+
| NAME | STATUS | RUNNER STATUS | POOL ID |
+-------------------------------------------+----------------+---------------+--------------------------------------+
| garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | pending_create | pending | fb25f308-7ad2-4769-988e-6ec2935f642a |
+-------------------------------------------+----------------+---------------+--------------------------------------+
```
We can also do a show on that runner to get more info:
```bash
ubuntu@experiments:~$ garm-cli runner show garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe
+-----------------+-------------------------------------------+
| FIELD | VALUE |
+-----------------+-------------------------------------------+
| ID | 089d63c9-5567-4318-a3a6-e065685c975b |
| Provider ID | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe |
| Name | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe |
| OS Type | linux |
| OS Architecture | amd64 |
| OS Name | ubuntu |
| OS Version | focal |
| Status | running |
| Runner Status | pending |
| Pool ID | fb25f308-7ad2-4769-988e-6ec2935f642a |
+-----------------+-------------------------------------------+
```
If we check out LXD, we can see the instance was created and is currently being bootstrapped:
```bash
ubuntu@experiments:~$ lxc list
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
| garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | RUNNING | 10.247.246.219 (enp5s0) | | VIRTUAL-MACHINE | 0 |
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
```
It might take a couple of minutes for the runner to come online, as the instance will do a full upgrade, then download the runner and install it. But once the installation is done you should see something like this:
```bash
ubuntu@experiments:~$ garm-cli runner show garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| FIELD | VALUE |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| ID | 089d63c9-5567-4318-a3a6-e065685c975b |
| Provider ID | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe |
| Name | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe |
| OS Type | linux |
| OS Architecture | amd64 |
| OS Name | ubuntu |
| OS Version | focal |
| Status | running |
| Runner Status | idle |
| Pool ID | fb25f308-7ad2-4769-988e-6ec2935f642a |
| Status Updates | 2022-05-06T13:21:54: downloading tools from https://github.com/actions/runner/releases/download/v2.291.1/actions-runner-linux-x64-2.291.1.tar.gz |
| | 2022-05-06T13:21:56: extracting runner |
| | 2022-05-06T13:21:58: installing dependencies |
| | 2022-05-06T13:22:07: configuring runner |
| | 2022-05-06T13:22:12: installing runner service |
| | 2022-05-06T13:22:12: starting service |
| | 2022-05-06T13:22:13: runner successfully installed |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
```
If we list the runners for this pool, we should see one runner with a ```RUNNER STATUS``` of ```idle```:
```bash
ubuntu@experiments:~$ garm-cli runner ls fb25f308-7ad2-4769-988e-6ec2935f642a
+-------------------------------------------+---------+---------------+--------------------------------------+
| NAME | STATUS | RUNNER STATUS | POOL ID |
+-------------------------------------------+---------+---------------+--------------------------------------+
| garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | running | idle | fb25f308-7ad2-4769-988e-6ec2935f642a |
+-------------------------------------------+---------+---------------+--------------------------------------+
```
## Updating a pool
Let's update the pool and request that it maintain a number of minimum idle runners equal to 3:
```bash
ubuntu@experiments:~$ garm-cli pool update fb25f308-7ad2-4769-988e-6ec2935f642a \
--min-idle-runners=3 \
--max-runners=10
+------------------+----------------------------------------------------------------------------------+
| FIELD | VALUE |
+------------------+----------------------------------------------------------------------------------+
| ID | fb25f308-7ad2-4769-988e-6ec2935f642a |
| Provider Name | lxd_local |
| Image | ubuntu:20.04 |
| Flavor | default |
| OS Type | linux |
| OS Architecture | amd64 |
| Max Runners | 10 |
| Min Idle Runners | 3 |
| Tags | ubuntu, simple-runner, repo-runner, self-hosted, x64, linux |
| Belongs to | gabriel-samfira/scripts |
| Level | repo |
| Enabled | true |
| Instances | garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe (089d63c9-5567-4318-a3a6-e065685c975b) |
+------------------+----------------------------------------------------------------------------------+
```
Now if we list runners we should see 2 more in ```pending``` state:
```bash
ubuntu@experiments:~$ garm-cli runner ls fb25f308-7ad2-4769-988e-6ec2935f642a
+-------------------------------------------+---------+---------------+--------------------------------------+
| NAME | STATUS | RUNNER STATUS | POOL ID |
+-------------------------------------------+---------+---------------+--------------------------------------+
| garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | running | idle | fb25f308-7ad2-4769-988e-6ec2935f642a |
+-------------------------------------------+---------+---------------+--------------------------------------+
| garm-bc180c6c-6e31-4c7b-8ce1-da0ffd76e247 | running | pending | fb25f308-7ad2-4769-988e-6ec2935f642a |
+-------------------------------------------+---------+---------------+--------------------------------------+
| garm-37c5daf4-18c5-47fc-95de-8c1656889093 | running | pending | fb25f308-7ad2-4769-988e-6ec2935f642a |
+-------------------------------------------+---------+---------------+--------------------------------------+
```
We can see them in LXC as well:
```bash
ubuntu@experiments:~$ lxc list
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
| garm-37c5daf4-18c5-47fc-95de-8c1656889093 | RUNNING | | | VIRTUAL-MACHINE | 0 |
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
| garm-bc180c6c-6e31-4c7b-8ce1-da0ffd76e247 | RUNNING | | | VIRTUAL-MACHINE | 0 |
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
| garm-edeb8f46-ab09-4ed9-88fc-2731ecf9aabe | RUNNING | 10.247.246.219 (enp5s0) | | VIRTUAL-MACHINE | 0 |
+-------------------------------------------+---------+-------------------------+------+-----------------+-----------+
```
Once they transition to ```idle```, you should see them in your repo settings, under ```Actions --> Runners```.
The procedure is identical for organizations. Have a look at the garm-cli help:
```bash
ubuntu@experiments:~$ garm-cli -h
CLI for the github self hosted runners manager.
Usage:
garm-cli [command]
Available Commands:
completion Generate the autocompletion script for the specified shell
credentials List configured credentials
debug-log Stream garm log
enterprise Manage enterprise
help Help about any command
init Initialize a newly installed garm
organization Manage organizations
pool List pools
profile Add, delete or update profiles
provider Interacts with the providers API resource.
repository Manage repositories
runner List runners in a pool
version Print version and exit
Flags:
--debug Enable debug on all API calls
-h, --help help for garm-cli
Use "garm-cli [command] --help" for more information about a command.
```

4
doc/the_boring_details.md Normal file
View file

@ -0,0 +1,4 @@
# GARM implementation details and design decissions
This document attempts to offer an in-depth look at the implementation details and design decisions that went into the creation of GARM. It is not meant to be a user guide, but rather a technical document for those interested in the inner workings of the application.

56
doc/webhooks.md Normal file
View file

@ -0,0 +1,56 @@
# Webhooks
Garm is designed to auto-scale github runners. To achieve this, ```garm``` relies on [GitHub Webhooks](https://docs.github.com/en/developers/webhooks-and-events/webhooks/about-webhooks). Webhooks allow ```garm``` to react to workflow events from your repository, organization or enterprise.
In your repository or organization, navigate to ```Settings --> Webhooks```:
![webhooks](images/webhooks.png)
And click on ```Add webhook```.
In the ```Payload URL``` field, enter the URL to the ```garm``` webhook endpoint. The ```garm``` API endpoint for webhooks is:
```txt
POST /webhooks
```
If ```garm``` is running on a server under the domain ```garm.example.com```, then that field should be set to ```https://garm.example.com/webhooks```.
In the webhook configuration page under ```Content type``` you will need to select ```application/json```, set the proper webhook URL and, really important, **make sure you configure a webhook secret**. Garm will authenticate the payloads to make sure they are coming from GitHub.
The webhook secret must be secure. Use something like this to generate one:
```bash
gabriel@rossak:~$ function generate_secret () {
tr -dc 'a-zA-Z0-9!@#$%^&*()_+?><~\`;' < /dev/urandom | head -c 64;
echo ''
}
gabriel@rossak:~$ generate_secret
9Q<fVm5dtRhUIJ>*nsr*S54g0imK64(!2$Ns6C!~VsH(p)cFj+AMLug%LM!R%FOQ
```
Make a note of that secret, as you'll need it later when you define the repo/org/enterprise in ```GARM```.
![webhook](images/input_url.png)
While you can use `http` for your webhook, I highly recommend you set up a proper x509 certificate for your GARM server and use `https` instead. If you choose `https`, GitHub will present you with an aditional option to configure the SSL certificate verification.
![ssl](images/tls_config.png)
If you're testing and want to use a self signed certificate, you can disable SSL verification or just use `http`, but for production you should use `https` with a proper certificate and SSL verification set to `enabled`.
It's fairly trivial to set up a proper x509 certificate for your GARM server. You can use [Let's Encrypt](https://letsencrypt.org/) to get a free certificate.
Next, you can choose which events GitHub should send to ```garm``` via webhooks. Click on ```Let me select individual events```.
![events](images/select_events.png)
Now select ```Workflow jobs``` (should be at the bottom). You can send everything if you want, but any events ```garm``` doesn't care about will simply be ignored.
![workflow](images/jobs.png)
Finally, click on ```Add webhook``` and you're done.
GitHub will send a test webhook to your endpoint. If all is well, you should see a green checkmark next to your webhook.

94
doc/webhooks_and_callbacks.md
View file

@ -1,94 +0,0 @@
# Webhooks
Garm is designed to auto-scale github runners based on a few simple rules:
* A minimum idle runner count can be set for a pool. Garm will attempt to maintain that minimum of idle runners, ready to be used by your workflows.
* A maximum number of runners for a pool. This is a hard limit of runners a pool will create, regardless of minimum idle runners.
* When a runner is scheduled by github, ```garm``` will automatically spin up a new runner to replace it, obeying the maximum hard limit defined.
To achieve this, ```garm``` relies on [GitHub Webhooks](https://docs.github.com/en/developers/webhooks-and-events/webhooks/about-webhooks). Webhooks allow ```garm``` to react to workflow events from your repository or organization.
In your repository or organization, navigate to ```Settings --> Webhooks```. In the ```Payload URL``` field, enter the URL to the ```garm``` webhook endpoint. The ```garm``` API endpoint for webhooks is:
```txt
POST /webhooks
```
If ```garm``` is running on a server under the domain ```garm.example.com```, then that field should be set to ```https://garm.example.com/webhooks```.
In the webhook configuration page under ```Content type``` you will need to select ```application/json```, set the proper webhook URL and, really important, **make sure you configure a webhook secret**. Garm will authenticate the payloads to make sure they are coming from GitHub.
The webhook secret must be secure. Use something like this to generate one:
```bash
gabriel@rossak:~$ function generate_secret () {
tr -dc 'a-zA-Z0-9!@#$%^&*()_+?><~\`;' < /dev/urandom | head -c 64;
echo ''
}
gabriel@rossak:~$ generate_secret
9Q<fVm5dtRhUIJ>*nsr*S54g0imK64(!2$Ns6C!~VsH(p)cFj+AMLug%LM!R%FOQ
```
Next, you can choose which events GitHub should send to ```garm``` via webhooks. Click on ```Let me select individual events``` and select ```Workflow jobs``` (should be at the bottom). You can send everything if you want, but any events ```garm``` doesn't care about will simply be ignored.
## The callback_url option
Your runners will call back home with status updates as they install. Once they are set up, they will also send the GitHub agent ID they were allocated. You will need to configure the ```callback_url``` option in the ```garm``` server config. This URL needs to point to the following API endpoint:
```txt
POST /api/v1/callbacks/status
```
Example of a runner sending status updates:
```bash
garm-cli runner show garm-f5227755-129d-4e2d-b306-377a8f3a5dfe
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| FIELD | VALUE |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| ID | 1afb407b-e9f7-4d75-a410-fc4a8c2dbe6c |
| Provider ID | garm-f5227755-129d-4e2d-b306-377a8f3a5dfe |
| Name | garm-f5227755-129d-4e2d-b306-377a8f3a5dfe |
| OS Type | linux |
| OS Architecture | amd64 |
| OS Name | ubuntu |
| OS Version | focal |
| Status | running |
| Runner Status | idle |
| Pool ID | 98f438b9-5549-4eaf-9bb7-1781533a455d |
| Status Updates | 2022-05-05T11:32:41: downloading tools from https://github.com/actions/runner/releases/download/v2.290.1/actions-runner-linux-x64-2.290.1.tar.gz |
| | 2022-05-05T11:32:43: extracting runner |
| | 2022-05-05T11:32:47: installing dependencies |
| | 2022-05-05T11:32:55: configuring runner |
| | 2022-05-05T11:32:59: installing runner service |
| | 2022-05-05T11:33:00: starting service |
| | 2022-05-05T11:33:00: runner successfully installed |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
```
This URL must be set and must be accessible by the instance. If you wish to restrict access to it, a reverse proxy can be configured to accept requests only from networks in which the runners ```garm``` manages will be spun up. This URL doesn't need to be globally accessible, it just needs to be accessible by the instances.
For example, in a scenario where you expose the API endpoint directly, this setting could look like the following:
```toml
callback_url = "https://garm.example.com/api/v1/callbacks/status"
```
Authentication is done using a short-lived JWT token, that gets generated for a particular instance that we are spinning up. That JWT token grants access to the instance to only update it's own status and to fetch metadata for itself. No other API endpoints will work with that JWT token. The validity of the token is equal to the pool bootstrap timeout value (default 20 minutes) plus the garm polling interval (5 minutes).
There is a sample ```nginx``` config [in the testdata folder](/testdata/nginx-server.conf). Feel free to customize it whichever way you see fit.
## The metadata_url option
The metadata URL is the base URL for any information an instance may need to fetch in order to finish setting itself up. As this URL may be placed behind a reverse proxy, you'll need to configure it in the ```garm``` config file. Ultimately this URL will need to point to the following ```garm``` API endpoint:
```bash
GET /api/v1/metadata
```
This URL needs to be accessible only by the instances ```garm``` sets up. This URL will not be used by anyone else. To configure it in ```garm``` add the following line in the ```[default]``` section of your ```garm``` config:
```toml
metadata_url = "https://garm.example.com/api/v1/metadata"
```

24
testdata/config.toml vendored
View file

@ -17,16 +17,15 @@ callback_url = "https://garm.example.com/api/v1/callbacks/status"
# highly encouraged.
metadata_url = "https://garm.example.com/api/v1/metadata"
# This folder is defined here for future use. Right now, we create a SSH
# public/private key-pair.
config_dir = "/etc/garm"
# Uncomment this line if you'd like to log to a file instead of standard output.
# log_file = "/tmp/runner-manager.log"
# Enable streaming logs via web sockets. Use garm-cli debug-log.
enable_log_streamer = false
# Enable the golang debug server. See the documentation in the "doc" folder for more information.
debug_server = false
[metrics]
# Toggle metrics. If set to false, the API endpoint for metrics collection will
# be disabled.
@ -78,34 +77,21 @@ time_to_live = "8760h"
debug = false
# Database backend to use. Currently supported backends are:
# * sqlite3
# * mysql
backend = "sqlite3"
# the passphrase option is a temporary measure by which we encrypt the webhook
# secret that gets saved to the database, using AES256. In the future, secrets
# will be saved to something like Barbican or Vault, eliminating the need for
# this. This setting needs to be 32 characters in size.
passphrase = "shreotsinWadquidAitNefayctowUrph"
[database.mysql]
# If MySQL is used, these are the credentials and connection information used
# to connect to the server instance.
# database username
username = ""
# Database password
password = ""
# hostname to connect to
hostname = ""
# database name
database = ""
[database.sqlite3]
# Path on disk to the sqlite3 database file.
db_file = "/etc/garm/garm.db"
# Currently, providers are defined statically in the config. This is due to the fact
# that we have not yet added support for storing secrets in something like Barbican
# or Vault. This will change in the future. However, for now, it's important to remember
# that once you create a pool using one of the providers defined here, the name of that
# provider must not be changes, or the pool will no longer work. Make sure you remove any
# provider must not be changed, or the pool will no longer work. Make sure you remove any
# pools before removing or changing a provider.
[[provider]]
# An arbitrary string describing this provider.
@ -135,7 +121,7 @@ time_to_live = "8760h"
# * virtual-machine (default)
# * container
#
instance_type = "virtual-machine"
instance_type = "container"
# enable/disable secure boot. If the image you select for the pool does not have a
# signed bootloader, set this to false, otherwise your instances won't boot.
secure_boot = false

4
util/appdefaults/appdefaults.go
View file

@ -35,10 +35,6 @@ const (
)
var (
// DefaultConfigDir is the default path on disk to the config dir. The config
// file will probably be in the same folder, but it is not mandatory.
DefaultConfigDir = "/etc/garm"
// DefaultUserGroups are the groups the default user will be part of.
DefaultUserGroups = []string{
"sudo", "adm", "cdrom", "dialout",