From ffb9d063a382ec66cada3ea3b2f479bef672c923 Mon Sep 17 00:00:00 2001 From: Daniel Sy Date: Wed, 19 Nov 2025 16:50:06 +0100 Subject: [PATCH] =?UTF-8?q?docs(forgejo):=20=F0=9F=92=84Updated=20docs=20a?= =?UTF-8?q?nd=20added=20diagrams?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- content/en/docs/components/forgejo/forgejo.md | 216 ++++++++++++------ go.mod | 1 + go.sum | 2 + hugo.toml | 2 + 4 files changed, 148 insertions(+), 73 deletions(-) diff --git a/content/en/docs/components/forgejo/forgejo.md b/content/en/docs/components/forgejo/forgejo.md index f363aea..dad005b 100644 --- a/content/en/docs/components/forgejo/forgejo.md +++ b/content/en/docs/components/forgejo/forgejo.md @@ -1,93 +1,163 @@ --- -title: "Forgejo Integration, Extension, and Community Collaboration" -linkTitle: Forgejo Software Forge -date: "2025-11-17" -description: "Summary of the project's work integrating GARM with Forgejo and contributing key features back to the community." -tags: ["Forgejo", "GARM", "CI/CD", "OSS", "Community", "Project Report"] +title: "Edge Developer Platform: Forgejo Integration & GARM Extension" +date: "2025-11-19" +description: "Technical report on the implementation of the Edge Developer Platform (EDP) using Forgejo, featuring GARM integration for ephemeral CI/CD runners and upstream open-source contributions." +tags: ["Forgejo", "GARM", "CI/CD", "OSS", "Architecture", "Project Report"] categories: ["Workpackage Results"] -weight: 10 +mermaid: true --- -{{% alert title="Draft" color="warning" %}} -**Editorial Status**: This page is currently being developed. +## 1. Project Identity & Governance -* **Jira Ticket**: [TICKET-6731](https://jira.telekom-mms.com/browse/IPCEICIS-6731) -* **Assignee**: Daniel -* **Status**: Draft -* **Last Updated**: 2025-11-17 -* **TODO**: - * [ ] Add concrete quick start steps - * [ ] Include prerequisites and access information - * [ ] Create first application tutorial -* **Review/Feedback**: - * [ ] Stephan: - * in general: - * [ ] some parts are worth to go th 'Governance' - * [ ] perhaps we should remove the emojis? - * [ ] perhaps we should avoid the impression that the text was copy/pated from AI - * some details/further ideas: - * [ ] where is it, this Forgejo? Why is it called 'edp.buildth.ing'? - * [ ] what are the components we use - package managament, actions, ... - * [ ] Friendly users? organisations? Public/private stuff? - * [ ] App Management discussions (we don't!)? - * [ ] what about code snippets how forgejo is deployed? SSO? user base? Federation options? - * [ ] storages, Redis, Postgres ... deployment options ... helm charts ... - * [ ] Migrations we did, where is the migration code? - * [ ] git POSIX filesystem concurrency discussion, S/3 bucket - * [ ] what is our general experience? - * [ ] repository centric domain data model - * [ ] how did we develop? which version did we take first? how did we upgrade? - * [ ] which development flows did we use? which pipleines? - * [ ] provide codeberg links for the PRs - * [ ] provide architecture drawings and repo links for the cache registry thing - * [ ] provide a hight level actions arch diagram from the perspective of forgejo - link to the GARM component here -{{% /alert %}} +### 1.1 Project Scope -## ๐Ÿงพ Result short description / cognitions +The internal service is officially designated as the **Edge Developer Platform (EDP)**. It is hosted at **[edp.buildth.ing](https://edp.buildth.ing)**. The domain selection followed a democratic team process to establish a unique identity distinct from standard corporate naming conventions. -Here is the management summary of the work package results: +**Access Model:** +The platform operates on a hybrid visibility model: -* **๐Ÿ“ˆ Strategic Selection:** We chose **[Forgejo](https://forgejo.org/)** as the project's self-hosted Git service. This decision was based on several key strategic factors: - * **EU-Based & Data Sovereignty:** The project is stewarded by **[Codeberg e.V.](https://docs.codeberg.org/getting-started/what-is-codeberg/)**, a non-profit based in Berlin, Germany. This is a massive win for our "funding agency" stakeholders, as it aligns with **GDPR, compliance, and data sovereignty goals**. It's governed by EU laws, not a US tech entity. - * **True Open Source (GPL v3+):** Forgejo is a community-driven fork of Gitea, created to *guarantee* it stays 100% free and open-source (FOSS). - * **License Protects Our Contributions:** It uses the **GPL v3+ "copyleft" license**. This is *perfect* for our collaboration goal. It legally ensures that the features we contribute back (like GARM support) can **never be taken and locked into a proprietary, closed-source product by anyone**. It protects our work and keeps the community open. +* **Public Access:** The [`DEVFW-CICD`](https://edp.buildth.ing/DevFW-CICD) organization is publicly accessible, fostering transparency. +* **Private Access:** Sensitive development occurs in restricted organizations (e.g., [`DEVFW`](https://edp.buildth.ing/DevFW)). +* **User Base:** Primary users include the internal development team, with friendly user access granted to the IPCEI team and MMS BT. -* **โš™๏ธ Core Use Case:** Forgejo is used for all project source code **versioning** and as the backbone for our **CI/CD (Continuous Integration/Continuous Deployment)** pipelines. +### 1.2 Strategic Selection & Governance -* **๐Ÿ› ๏ธ Key Extension (GARM Support):** The main technical achievement was integrating **[GARM (GitHub Actions Runner Manager)](https://github.com/cloudbase/garm)**. This was *not* supported by Forgejo out-of-the-box. +The decision to utilize **[Forgejo](https://forgejo.org/)** as the core self-hosted Git service was driven by specific strategic requirements: -* **โœจ Required Enhancements:** To make GARM work, our team developed and implemented several critical features: - * Webhook support for workflow events (to tell runners when to start). - * Support for ephemeral runners (for secure, clean-slate builds every time). - * GitHub API-compatible endpoints (to allow the runners to register themselves correctly). - -* **๐Ÿ’– Community Contribution:** We didn't just keep this for ourselves! We contributed all these features **directly back to the upstream Forgejo community**. This wasn't just a code-dump; we actively collaborated via **issues**, **feature requests**, and **pull requests (PRs) on [codeberg.org](https://codeberg.org/)**. - -* **๐Ÿš€ Bonus Functionality:** We also implemented **artifact caching**. This configures Forgejo to act as a **pull-through proxy** for remote container registries (like Docker Hub), which seriously speeds up our build times and saves bandwidth. +* **EU-Based Stewardship:** Forgejo is stewarded by **[Codeberg e.V.](https://docs.codeberg.org/getting-started/what-is-codeberg/)**, a non-profit organization based in Berlin, Germany. This alignment ensures compliance with GDPR and data sovereignty requirements, placing governance under EU jurisdiction rather than US tech entities. +* **License Protection (GPL v3+):** Unlike "Open Core" models, Forgejo uses a copyleft license. This legally protects our custom extensions (such as GARM support) from being appropriated into proprietary software, ensuring the ecosystem remains open. +* **Open Source Strategy:** The platform aligns with the "Public Money, Public Code" philosophy, mandating that funded developments are returned to the community. --- -## ๐Ÿ“Š Key Performance Indicators (KPIs) +## 2. Technical Architecture & Deployment -These metrics assess the success of the work package across technical quality, development efficiency, and open-source engagement. +### 2.1 Infrastructure Stack -### ๐Ÿš€ CI/CD & Delivery Efficiency +The platform is hosted on the **Open Telekom Cloud (OTC)**. The infrastructure adheres to Infrastructure-as-Code (IaC) principles. -These KPIs measure the effectiveness of the Forgejo + GARM setup. +* **Deployment Method:** The official Forgejo Helm Chart is deployed via **ArgoCD**. +* **Infrastructure Provisioning:** **Terraform** is used to provision all underlying OTC services, including: + * **Container Orchestration**: CCE (Cloud Container Engine): Kubernetes + * **Database:** RDS (Distributed Cache Service): PostgreSQL + * **Caching:** DCS (Distributed Cache Service): Redis + * **Object Storage:** OBS (Object Storage Service, S3-compatible): for user data (avatars, attachments). + * **Search:** CSS (Cloud Search Service): Elasticsearch + +### 2.2 The "Self-Replicating" Pipeline + +A key architectural feature is the ability of the platform to maintain itself. A Forgejo Action can trigger the deployment script, which runs Terraform and syncs ArgoCD, effectively allowing "Forgejo to create/update Forgejo." + +```mermaid +graph TD + subgraph "Open Telekom Cloud (OTC)" + subgraph "Control Plane" + Dev[DevOps Engineer] -->|Triggers| Pipeline[Deployment Pipeline] + Pipeline -->|Executes| TF[Terraform] + end + + subgraph "Provisioned Infrastructure" + TF -->|Provisions| CCE[(CCE K8s Cluster)] + TF -->|Provisions| RDS[(RDS PostgreSQL)] + TF -->|Provisions| Redis[(DCS Redis)] + TF -->|Provisions| S3[(OBS S3 Bucket)] + TF -->|Provisions| CSS[(CSS Elasticsearch)] + end + + subgraph "Application Layer (on CCE K8s)" + Pipeline -->|Helm Chart| Argo[ArgoCD] + Argo -->|Deploys| ForgejoApp[Forgejo] + end + + CCE -- Runs --> Argo + CCE -- Runs --> ForgejoApp + ForgejoApp -->|Connects| RDS + ForgejoApp -->|Connects| Redis + ForgejoApp -->|Connects| S3 + ForgejoApp -->|Connects| CSS + end +``` + +### 2.3 Migration History + +The initial environment was a manual setup on the Open Sovereign Cloud (OSC). Once the automation stack (Terraform/ArgoCD) was matured, the platform was migrated to the current OTC environment. + +--- + +## 3. Application Extensions & GARM Integration + +### 3.1 Core Functionality + +Beyond standard Git versioning, the platform utilizes: + +* **Releases:** Hosting binaries for software distribution (e.g., Edge Connect CLI). +* **CI/CD:** Extensive pipeline usage for build, test, and deployment automation. +* **Note on Issues:** While initially used, issue tracking was migrated to JIRA to align with the broader IPCEI program standards. + +### 3.2 GARM (Git-based Actions Runner Manager) + +The primary technical innovation was the integration of **[GARM](https://github.com/cloudbase/garm)** to enable ephemeral, scalable runners. This required extending Forgejo's capabilities to support GitHub-compatible runner registration and webhook events. + +**Workflow Architecture:** + +1. **Event:** A workflow event occurs in Forgejo. +2. **Trigger:** A webhook notifies GARM. +3. **Provisioning:** GARM spins up a fresh, ephemeral runner. +4. **Execution:** The runner registers via the API, executes the job, and is terminated immediately after, ensuring a clean build environment. + +```mermaid +sequenceDiagram + participant User + participant Forgejo + participant GARM + participant Runner as Ephemeral Runner + + User->>Forgejo: Push Code / Trigger Event + Forgejo->>GARM: Webhook Event (Workflow Dispatch) + GARM->>Forgejo: Register Runner (via API) + GARM->>Runner: Spin up Instance + Runner->>Forgejo: Request Job + Forgejo->>Runner: Send Job Payload + Runner->>Runner: Execute Steps + Runner->>Forgejo: Report Status + GARM->>Runner: Terminate (Ephemeral) +``` + +--- + +## 4. Development Methodology & Contributions + +### 4.1 Workflow + +* **Branching Strategy:** Trunk-based development was utilized to ensure rapid integration. +* **Collaboration:** The team adopted **Mob Programming**. This practice proved essential for knowledge sharing and onboarding junior developers, creating a resilient and high-intensity learning environment. +* **Versions:** The platform evolved from Forgejo v7/8 to the current v11.0.3-edp1. An upgrade is pending to leverage the latest upstream GARM features. + +### 4.2 Open Source Contributions + +We actively contributed our extensions back to the upstream Forgejo project on **[Codeberg.org](https://codeberg.org/)**. + +**Key Pull Requests:** + +* **API Compatibility:** Added GitHub-compatible endpoints for runner registration. + * [PR #9409: Feat: Add endpoints for GARM](https://codeberg.org/forgejo/forgejo/pulls/9409) +* **Webhook Support:** Implemented webhook triggers for workflow events. + * [PR #9803: Feat: Add webhook support for workflow events](https://codeberg.org/forgejo/forgejo/pulls/9803) +* **Ephemeral Runners:** Added support for runners that terminate after a single job. + * [PR #9962: Feat: Support for ephemeral runners](https://codeberg.org/forgejo/forgejo/pulls/9962) + +**Artifact Caching (Pull-Through Proxy):** +We implemented a feature allowing Forgejo to act as a pull-through proxy for remote container registries, optimizing bandwidth and build speeds. + +* [Source Code Branch: refactor-remote-registry-client](https://edp.buildth.ing/DevFW/edp-forgejo/src/branch/refactor-remote-registry-client) + +--- + +## 5. Key Performance Indicators (KPIs) | KPI | Description | Target / Benchmark | | :--- | :--- | :--- | -| **Deployment Frequency** | How often changes are successfully pushed to production/staging (e.g., deploys per day/week). | High frequency (daily or better) indicates high agility. | -| **Change Failure Rate (CFR)** | Percentage of deployments that require immediate remediation (e.g., a rollback or hotfix). | Below 5% (Industry standard for high performance). | -| **Cycle Time** | Time from the first commit to deployment/release. | Measured in days, trending downward, indicating pipeline efficiency. | -| **Artifact Cache Hit Rate** | Percentage of build requests successfully served by the local Forgejo proxy cache. | **90%+** demonstrates effective use of the proxy and reduced external dependencies. | - -### ๐Ÿ’– Open Source & Community Engagement - -These KPIs quantify our strategic commitment to the Forgejo community. - -| KPI | Description | Target / Benchmark | -| :--- | :--- | :--- | -| **Upstream Contribution Ratio** | Percentage of project code (GARM-related features) committed to Forgejo's upstream repositories versus being maintained in a private fork. | **100%** demonstrates full commitment to collaboration. | -| **Pull Request (PR) Resolution Time** | Average time taken for a PR submitted to Codeberg to be merged or closed by the Forgejo community. | Trending downward (e.g., < 7 days) indicates healthy, responsive collaboration. | -| **Number of Accepted External PRs** | Count of community-submitted features/fixes related to our GARM extension that we review/merge into our implementation. | Increase over time indicates growing external adoption/interest in our work. | +| **Deployment Frequency** | Frequency of successful pipeline executions. | High (Daily/On-demand) | +| **Artifact Cache Hit Rate** | Percentage of build requests served by the local Forgejo proxy. | > 90% (Reduced external traffic) | +| **Upstream Contribution** | Percentage of GARM-related features contributed back to Codeberg. | 100% (No vendor lock-in) | +| **PR Resolution Time** | Average time for upstream community review and merge. | < 14 days (Healthy collaboration) | diff --git a/go.mod b/go.mod index f4c0b45..9cd543d 100644 --- a/go.mod +++ b/go.mod @@ -5,5 +5,6 @@ go 1.22.5 require ( github.com/FortAwesome/Font-Awesome v0.0.0-20241216213156-af620534bfc3 // indirect github.com/google/docsy v0.12.0 // indirect + github.com/hugomods/mermaid v0.1.4 // indirect github.com/twbs/bootstrap v5.3.8+incompatible // indirect ) diff --git a/go.sum b/go.sum index 6a51e65..a5d6dc2 100644 --- a/go.sum +++ b/go.sum @@ -2,6 +2,8 @@ github.com/FortAwesome/Font-Awesome v0.0.0-20241216213156-af620534bfc3 h1:/iluJk github.com/FortAwesome/Font-Awesome v0.0.0-20241216213156-af620534bfc3/go.mod h1:IUgezN/MFpCDIlFezw3L8j83oeiIuYoj28Miwr/KUYo= github.com/google/docsy v0.12.0 h1:CddZKL39YyJzawr8GTVaakvcUTCJRAAYdz7W0qfZ2P4= github.com/google/docsy v0.12.0/go.mod h1:1bioDqA493neyFesaTvQ9reV0V2vYy+xUAnlnz7+miM= +github.com/hugomods/mermaid v0.1.4 h1:/u0FZRSBMpqTbDh4XPBL7c6p0DUjvIkoo9cC89GASSI= +github.com/hugomods/mermaid v0.1.4/go.mod h1:GOID2Ko1vdrX03ZMaFNQL/vdTU1tZmGcVjOgBN8O42Y= github.com/twbs/bootstrap v5.3.6+incompatible/go.mod h1:fZTSrkpSf0/HkL0IIJzvVspTt1r9zuf7XlZau8kpcY0= github.com/twbs/bootstrap v5.3.8+incompatible h1:eK1fsXP7R/FWFt+sSNmmvUH9usPocf240nWVw7Dh02o= github.com/twbs/bootstrap v5.3.8+incompatible/go.mod h1:fZTSrkpSf0/HkL0IIJzvVspTt1r9zuf7XlZau8kpcY0= diff --git a/hugo.toml b/hugo.toml index 9b98038..62c15a4 100644 --- a/hugo.toml +++ b/hugo.toml @@ -156,6 +156,8 @@ enable = false [[module.imports]] path = "github.com/google/docsy" disable = false +[[module.imports]] +path = "github.com/hugomods/mermaid" [params.mermaid] version = "10.9.0"