initial commit

Signed-off-by: salaboy <Salaboy@gmail.com>
2024-08-02 10:07:46 +01:00 · 2024-08-02 10:07:46 +01:00 · c12b696293
commit c12b696293
parent efa49b74ea
7 changed files with 496 additions and 11 deletions
--- a/README.md
+++ b/README.md
@ -1,18 +1,356 @@
-# CNOE Stacks
+[![Codespell][codespell-badge]][codespell-link]
+[![E2E][e2e-badge]][e2e-link]
+[![Go Report Card][report-badge]][report-link]
+[![Commit Activity][commit-activity-badge]][commit-activity-link]

-This repository contains building blocks and examples to help you build your own Internal Developer Platform.
+# IDP Builder
+
+Internal development platform binary launcher.
+
+> **WORK IN PROGRESS**: This tool is in a pre-release stage and is under active development.
+
+## About
+
+Spin up a complete internal developer platform using industry standard technologies like Kubernetes, Argo, and backstage with only Docker required as a dependency.
+
+This can be useful in several ways:
+* Create a single binary which can demonstrate an IDP reference implementation.
+* Use within CI to perform integration testing.
+* Use as a local development environment for IDP engineers.
+
+## Prerequisites
+
+A container engine is needed locally such as:
+
+| Name                                                  | Supported | Remark                                                                                                                              |
+|-------------------------------------------------------|-----------|-------------------------------------------------------------------------------------------------------------------------------------|
+| [Docker desktop](https://www.docker.com/get-started/) | Yes       |                                                                                                                                     |
+| [Podman desktop](https://podman-desktop.io/)          | No        | idpbuilder can create a cluster using podman [rootful](https://docs.podman.io/en/latest/markdown/podman-machine-set.1.html#rootful) | 
+
+
+**Note**: Set the `DOCKER_HOST` env var property using `podman` to let idpbuilder to talk with the engine (e.g  export DOCKER_HOST="unix:///var/run/docker.sock")

 ## Getting Started

-### Install idpbuilder
+If you are interested in running idpbuilder in Codespaces through your browser, check out the [Codespaces](#running-in-codespaces) section.

-To get started, you need to install [`idpbuilder`](https://github.com/cnoe-io/idpbuilder).
-See [the instructions](https://github.com/cnoe-io/idpbuilder?tab=readme-ov-file#getting-started) in the idpbuilder repository for details.
+### Quick Install

-### Using this repository
+The following command can be used as a convenience for installing `idpbuilder`, (be sure to check the script first if you are concerned):
+```
+curl -fsSL https://raw.githubusercontent.com/cnoe-io/idpbuilder/main/hack/install.sh | bash
+```

- **[CNOE Reference Implementation](./ref-implementation)**. Create a local CNOE environment in minutes. 
- **[Basic Examples](./basic)**. Do you want to know how to use idpbuilder with basic examples?
- **[Local Backup](./local-backup)**. How do I make sure my work is backed up?
- **[Localstack](./localstack-integration)**. Use [LocalStack](https://github.com/localstack/localstack) to test out cloud integrations.
- **[Terraform Integrations](./terraform-integrations)**. Integrating Terraform with Reference Implementation.
+or download the latest stable release with the following commands:
+
+```bash
+version=$(curl -s https://api.github.com/repos/cnoe-io/idpbuilder/releases | grep tag_name | grep -o -e '"v[0-9].[0-9].[0-9]"' | head -n1 | sed 's/"//g')
+curl -L --progress-bar -o ./idpbuilder.tar.gz "https://github.com/cnoe-io/idpbuilder/releases/download/${version}/idpbuilder-$(uname | awk '{print tolower($0)}')-$(uname -m | sed 's/x86_64/amd64/').tar.gz"
+tar xzf idpbuilder.tar.gz
+
+./idpbuilder version
+# example output
+# idpbuilder 0.4.1 go1.21.5 linux/amd64
+```
+
+Alternatively, you can download the latest binary from [the latest release page](https://github.com/cnoe-io/idpbuilder/releases/latest).
+
+## Using the idpbuilder
+
+### Basic usage
+
+The most basic command which creates a Kubernetes Cluster (Kind cluster) with the core packages installed.
+
+```bash
+./idpbuilder create
+```
+
+<details>
+  <summary>What are the core packages?</summary>
+
+  * **ArgoCD** is the GitOps solution to deploy manifests to Kubernetes clusters. In this project, a package is an ArgoCD application. 
+  * **Gitea** server is the in-cluster Git server that ArgoCD can be configured to sync resources from. You can sync from local file systems to this.
+  * **Ingress-nginx** is used as a method to access in-cluster resources such as ArgoCD UI and Gitea UI.
+
+    #### Core package versions
+    
+    | Name     | Version |
+    | -------- | ------- |
+    | Argo CD  | v2.10.7 |
+    | Gitea    | v9.5.1  |
+    | Nginx    | v1.8.1  |
+
+  The default manifests for the core packages are available [here](pkg/controllers/localbuild/resources).
+  See the [contribution doc](./CONTRIBUTING.md) for more information on how core packages are installed and configured.
+
+</details>
+
+
+Once idpbuilder finishes provisioning cluster and packages, you can access GUIs by going to the following addresses in your browser.
+
+* ArgoCD: https://argocd.cnoe.localtest.me:8443/
+* Gitea: https://gitea.cnoe.localtest.me:8443/
+
+#### Secrets
+You can obtain credentials for them by running the following command:
+
+```bash
+./idpbuilder get secrets
+```
+
+<details>
+  <summary> The "get secrets" command </summary>
+
+  The `get secrets` command retrieves the following:
+  - ArgoCD initial admin password.
+  - Gitea admin user credentials.
+  -  Any secrets labeled with `cnoe.io/cli-secret=true`.
+
+  You can think of the command as executing the following kubectl commands:
+
+  ```bash
+  kubectl -n argocd get secret argocd-initial-admin-secret
+  kubectl get secrets -n gitea gitea-admin-secret
+  kubectl get secrets -A -l cnoe.io/cli-secret=true
+  ```
+  In addition, secrets labeled with `cnoe.io/package-name` can be specified with the `-p` flag. For example, for Gitea:
+
+  ```bash
+  ./idpbuilder get secrets -p gitea
+  ```
+
+</details>
+
+###  Example commands
+
+**For more advanced use cases, check out the [Stacks Repository](https://github.com/cnoe-io/stacks).**
+
+You can specify the kubernetes version by using the `--kube-version` flag. Supported versions are available [here](https://github.com/kubernetes-sigs/kind/releases).
+
+```
+./idpbuilder create --kube-version v1.27.3
+```
+
+If you want to specify your own kind configuration file, use the `--kind-config` flag.
+
+```
+./idpbuilder create --build-name local --kind-config ./my-kind.yaml`
+```
+
+If you want to specify ArgoCD configmap.
+
+```
+./idpbuilder create --package-custom-file=argocd:pkg/k8s/test-resources/input/argocd-cm.yaml
+```
+
+Run the following commands for available flags and subcommands:
+
+```
+./idpbuilder --help
+./idpbuilder create --help
+```
+
+### Custom Packages
+
+Idpbuilder supports specifying custom packages using the flag `--package` flag. 
+This flag expects a directory (local or remote) containing ArgoCD application files and / or ArgoCD application set files.
+In case of a remote directory, it must be a directory in a git repository, 
+and the URL format must be a [kustomize remote URL format](https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md).
+
+Examples of using custom packages are available in the [stacks repository](https://github.com/cnoe-io/stacks).
+Let's take a look at [this example](https://github.com/cnoe-io/stacks/tree/main/basic). This defines two custom package directories to deploy to the cluster.
+
+To deploy these packages, run the following command.
+
+```
+idpbuilder create --package https://github.com/cnoe-io/stacks//basic/package1 --package https://github.com/cnoe-io/stacks//basic/package2
+
+```
+
+Alternatively, you can use the local directory format.
+
+```bash
+# clone the stacks repository
+git clone https://github.com/cnoe-io/stacks.git
+cd stacks
+# run idpbuilder against the local directory
+idpbuilder create --package basic/package1  --package basic/package2
+
+```
+
+Running this command should create three additional ArgoCD applications in your cluster.
+
+```sh
+$ kubectl get Applications -n argocd  -l example=basic
+NAME         SYNC STATUS   HEALTH STATUS
+guestbook    Synced        Healthy
+guestbook2   Synced        Healthy
+my-app       Synced        Healthy
+```
+
+Let's break this down. The [first package directory](https://github.com/cnoe-io/stacks/tree/main/basic/package1) defines an application. 
+This corresponds to the `my-app` application above. In this application, we want to deploy manifests from local machine in GitOps way.
+
+The directory contains an [ArgoCD application file](https://github.com/cnoe-io/stacks/blob/main/basic/package1/app.yaml). 
+This is a normal ArgoCD application file except for one field.
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+spec:
+  source:
+    repoURL: cnoe://manifests
+```
+
+The `cnoe://` prefix in the `repoURL` field indicates that we want to sync from a local directory.
+Values after `cnoe://` is treated as a relative path from this file. In this example, 
+we are instructing idpbuilder to make ArgoCD sync from files in the [manifests directory](https://github.com/cnoe-io/stacks/tree/main/basic/package1/manifests).
+
+As a result the following actions were taken by idpbuilder: 
+1. Create a Gitea repository.
+2. Fill the repository with contents from the manifests directory.
+3. Update the Application spec to use the newly created repository.
+
+You can verify this by going to this address in your browser: https://gitea.cnoe.localtest.me:8443/giteaAdmin/idpbuilder-localdev-my-app-manifests
+
+![img.png](docs/images/my-app-repo.png)
+
+
+This is the repository that corresponds to the [manifests](https://github.com/cnoe-io/stacks/tree/main/basic/package1/manifests) folder.
+It contains a file called `alpine.yaml`, synced from the `manifests` directory above.
+
+You can also view the updated Application spec by going to this address: https://argocd.cnoe.localtest.me:8443/applications/argocd/my-app
+
+![myapp](docs/images/my-app.png)
+
+
+The second package directory defines two normal ArgoCD applications referencing a remote repository.
+They are applied as-is.
+
+## Local OCI Registry
+
+The local Gitea instance created by idpbuilder contains a built in OCI registry for hosting container images as "packages" in Gitea nomenclature.
+
+It is a standard OCI registry, so the API should be compatible with any tools that are OCI compliant. That includes the `docker` cli.
+
+For example you can push an image by running:
+
+```bash
+docker login gitea.cnoe.localtest.me:8443                                          
+Username: giteaAdmin
+Password: 
+docker push gitea.cnoe.localtest.me:8443/giteaadmin/beacon.idpbuilder:with-app-fix2
+The push refers to repository [gitea.cnoe.localtest.me:8443/giteaadmin/beacon.idpbuilder]
+78a0cd9d2976: Layer already exists 
+with-app-fix2: digest: sha256:50dc814b89e22988a69ac23aa7158daa834ab450b38b299e7f7fe17dba0ce992 size: 5566
+```
+
+*NOTE: You can get the giteaAdmin password in the same way as you do for the web or git interface.*
+
+```bash
+  ./idpbuilder get secrets -p gitea
+```
+
+Or you can use this one liner to login:
+
+```bash
+idpbuilder get secrets -p gitea -o json | jq '.[0].data.password' -r | docker login -u giteaAdmin --password-stdin gitea.cnoe.localtest.me:8443
+```
+
+### Pulling Images
+
+You can pull an image back to your local machine using your docker client like so:
+
+```
+docker push gitea.cnoe.localtest.me:8443/giteaadmin/beacon.idpbuilder
+Using default tag: latest
+latest: Pulling from giteaadmin/beacon.idpbuilder
+Digest: sha256:6308ebbce176470277dcca5e59aee3d528d9798a19f13d6a73ddd74a3f5da17b
+Status: Downloaded newer image for gitea.cnoe.localtest.me:8443/giteaadmin/beacon.idpbuilder:latest
+gitea.cnoe.localtest.me:8443/giteaadmin/beacon.idpbuilder:latest
+```
+
+### Referencing Images In Manifests On The Idpbuilder K8s Cluster
+If you are creating a pod or a deployment of some sort, you can reference the images on the cluster using the same image name and tag like in the following example:
+
+```
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  annotations:
+spec:
+  template:
+    spec:
+      containers:
+      - image: gitea.cnoe.localtest.me:8443/giteaadmin/beacon.idpbuilder:with-app-fix2
+        imagePullPolicy: IfNotPresent
+```
+
+### No Pull Secret Needed
+Our gitea instance allows for anonymous read access. This means that you can pull git repo contents and container images without the need to login.
+
+### Only Works With Subdomain Based Idpbuilder Installations
+Right now because of the way the OCI registry specifications discovers information about a repo, this will only work with subdomain `gitea.cnoe.localtest.me`
+based installations of idpbuilder's core capabilities.
+
+If you would like to use path based routing, you will have to install and manage your own OCI registry at this time.
+Other registries might be able to handle this better, however which registries and how to configure them is beyond the scope of this readme.
+
+For more info on the OCI registry spec and the root cause of this "discovery" issue see the spec here:
+https://specs.opencontainers.org/distribution-spec/?v=v1.0.0#checking-if-content-exists-in-the-registry
+
+### Pulling Images From Inside Idpbuilder K8s Cluster:
+
+Because we are using an NGINX Ingress and pushing our image from off cluster,
+Gitea and it's OCI registry think all images pushed to it are prefixed with `gitea.cnoe.localtest.me:8443`.
+
+This is correct by the OCI spec standards. However when you are on the cluster, that ingress is not available to you. 
+You can use the service name of gitea, but gitea will not know what images are being asked for at the svc domain name.
+
+So we use containerd to rewrite those image names so that they can be referenced at the external url:
+
+See `./pkg/kind/resources/kind.yaml.tmpl` for how this is done.
+
+## Contributing
+
+If you'd like to contribute to the project or know the architecture and internals of this project, check out the [contribution doc](./CONTRIBUTING.md).
+
+## Running in Codespaces
+
+1. Create a Codespaces instance. ![img](https://github.com/cnoe-io/stacks/blob/main/ref-implementation/images/codespaces-create.png)
+2. Wait for it to be ready. It may take several minutes.
+3. Get the latest release of idpbuilder:
+   ```bash
+    version=$(curl -Ls -o /dev/null -w %{url_effective} https://github.com/cnoe-io/idpbuilder/releases/latest)
+    version=${version##*/}
+    curl -L -o ./idpbuilder.tar.gz "https://github.com/cnoe-io/idpbuilder/releases/download/${version}/idpbuilder-$(uname | awk '{print tolower($0)}')-$(uname -m | sed 's/x86_64/amd64/').tar.gz"
+    tar xzf idpbuilder.tar.gz
+   ```
+4. Run idpbuilder:
+   ```
+    idpbuilder create --protocol http  \
+    --host ${CODESPACE_NAME}-8080.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN} \
+    --port 8080 --use-path-routing
+   ```
+5. Because Codespaces gives a single externally routable host name for an instance, idpbuilder must deploy with path based routing. 
+   This means ArgoCD and Gitea UIs are given with the following commands.
+   * ArgoCD: `echo https://${CODESPACE_NAME}-8080.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN}/argocd`
+   * Gitea: `echo https://${CODESPACE_NAME}-8080.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN}/gitea`
+6. Note that not all examples work with path based routing. 
+
+## Extending the IDP builder
+
+We are actively working to include more patterns and examples of extending idpbuilder to get started easily.
+
+<!-- JUST BADGES & LINKS -->
+[codespell-badge]: https://github.com/cnoe-io/idpbuilder/actions/workflows/codespell.yaml/badge.svg
+[codespell-link]: https://github.com/cnoe-io/idpbuilder/actions/workflows/codespell.yaml
+
+[e2e-badge]: https://github.com/cnoe-io/idpbuilder/actions/workflows/e2e.yaml/badge.svg
+[e2e-link]: https://github.com/cnoe-io/idpbuilder/actions/workflows/e2e.yaml
+
+[report-badge]: https://goreportcard.com/badge/github.com/cnoe-io/idpbuilder
+[report-link]: https://goreportcard.com/report/github.com/cnoe-io/idpbuilder
+
+[commit-activity-badge]: https://img.shields.io/github/commit-activity/m/cnoe-io/idpbuilder
+[commit-activity-link]: https://github.com/cnoe-io/idpbuilder/pulse
--- a/dapr-integration/README.md
+++ b/dapr-integration/README.md
@ -0,0 +1,36 @@
+# Dapr Integrations 
+
+`idpBuilder` is extensible to launch custom Dapr patterns using package extensions. 
+
+Please use the below command to deploy an IDP reference implementation with an Argo application for preparing up the setup for terraform integrations:
+
+```bash
+idpbuilder create \
+  --use-path-routing \
+  --package-dir https://github.com/cnoe-io/stacks//ref-implementation \
+  --package-dir https://github.com/cnoe-io/stacks//dapr-integrations
+```
+## What is installed?
+
+1. Dapr Control Plane
+1. Dapr Statestore and PubSub components
+
+This needs your credentials for this to work. Follow the Dapr installation documentation on how to add your credentials.
+
+## Application with cloud resources.
+
+With this integration, we can deploy an application with cloud resources using Backstage templates from the reference implementation, together with Dapr integrations.
+
+In this example, we will create an application with a S3 Bucket.
+
+Choose a template named `App with S3 bucket`, type `demo3` as the name, then choose a region to create this bucket in.
+
+Once you click the create button, you will have a very similar setup as the basic example.
+The only difference is we now have a resource for a S3 Bucket which is managed by Crossplane.
+
+Note that Bucket is **not** created because Crossplane doesn't have necessary credentials to do so.
+If you'd like it to actually create a bucket, update [the credentials secret file](crossplane-providers/provider-secret.yaml), then run `idpbuilder create --package-dir examples/ref-implementation`.
+
+In this example, we used Crossplane to provision resources, but you can use other cloud resource management tools such as Terraform instead.
+
+Regardless of your tool choice, concepts are the same. We use Backstage as the templating mechanism and UI for users, then use Kubernetes API with GitOps to deploy resources.
--- a/dapr-integration/dapr-components.yaml
+++ b/dapr-integration/dapr-components.yaml
@ -0,0 +1,22 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: dapr-components
+  namespace: argocd
+  labels:
+    env: dev
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: default
+  source:
+    repoURL: cnoe://dapr-components/manifests
+    targetRevision: HEAD
+    path: "."
+    directory:
+      recurse: true
+  destination:
+    server: "https://kubernetes.default.svc"
+    namespace: default
+  syncPolicy:
+    automated: {}
--- a/dapr-integration/dapr-components/pubsub.yaml
+++ b/dapr-integration/dapr-components/pubsub.yaml
@ -0,0 +1,21 @@
+apiVersion: dapr.io/v1alpha1
+kind: Component
+metadata:
+  name: pubsub
+spec:
+  type: pubsub.redis
+  version: v1
+  metadata:
+    # These settings will work out of the box if you use `helm install
+    # bitnami/redis`.  If you have your own setup, replace
+    # `redis-master:6379` with your own Redis master address, and the
+    # Redis password with your own Secret's name. For more information,
+    # see https://docs.dapr.io/operations/components/component-secrets .
+    - name: redisHost
+      value: redis-master:6379
+    - name: redisPassword
+      secretKeyRef:
+        name: redis
+        key: redis-password
+auth:
+  secretStore: kubernetes
--- a/dapr-integration/dapr-components/statestore.yaml
+++ b/dapr-integration/dapr-components/statestore.yaml
@ -0,0 +1,16 @@
+apiVersion: dapr.io/v1alpha1
+kind: Component
+metadata:
+  name: statestore
+spec:
+  type: state.redis
+  version: v1
+  metadata:
+    - name: redisHost
+      value: redis-master:6379
+    - name: redisPassword
+      secretKeyRef:
+        name: redis
+        key: redis-password
+auth:
+  secretStore: kubernetes
--- a/dapr-integration/dapr.yaml
+++ b/dapr-integration/dapr.yaml
@ -0,0 +1,26 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: dapr
+  namespace: argocd
+  labels:
+    env: dev
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: default
+  source:
+    repoURL: 'https://dapr.github.io/helm-charts/'
+    targetRevision: 1.13.5
+    helm:
+      releaseName: dapr
+    chart: dapr
+  destination:
+    server: 'https://kubernetes.default.svc'
+    namespace: dapr-system
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
--- a/dapr-integration/redis.yaml
+++ b/dapr-integration/redis.yaml
@ -0,0 +1,26 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: redis-dapr
+  namespace: argocd
+  labels:
+    env: dev
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: default
+  source:
+    repoURL: 'oci://registry-1.docker.io/bitnamicharts'
+    targetRevision: 19.6.4
+    helm:
+      releaseName: redis
+    chart: redis
+  destination:
+    server: 'https://kubernetes.default.svc'
+    namespace: default
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true