ai-workspace-infra/iac_modules
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
c2020da184
iac_modules/README.md
Haitao Pan c2020da184 feat(iac): Refactor structure and support multi-environment config loading 
- Add config/sit and other environment-specific config directories
- Refactor deploy.py to support CONFIG_PATH environment variable
- Enable automatic merging of config/*/*.yaml files
- Enhance run.sh with Pulumi/Ansible/Terraform initialization checks
- Add inventory.py to dynamically generate Ansible hosts
- Improve ec2_instance.py with modular instance creation
- Organize base.yaml, vpc.yaml and related config files"
2025-03-29 11:09:24 +08:00

11 KiB
Raw Blame History

Modern Container Application Reference Architecture

Welcome to the repository for the Modern Container Application Reference Architecture. This repository contains a comprehensive guide and reference architecture for building scalable, portable, resilient, and agile containerized applications. 一个基于 Pulumi + Ansible 的基础设施自动化项目模板，支持多环境部署（dev / staging / prod），实现从基础设施创建到主机配置的全流程自动化管理。

Overview

The project aims to create a multi-cloud environment that leverages containers for deploying modern applications. The key objective is to set up a unified authentication system using OIDC via Keycloak for AWS, GCP, Azure, GitHub, **Harbor ** and **Grafana **.

🚀 项目功能

  • 使用 Pulumi（Python）创建 AWS 基础设施（VPC、子网、安全组、EC2）
  • 配置结构模块化：base.yaml, vpc.yaml, firewall.yaml, instances.yaml
  • 支持 Spot / On-Demand 实例，支持 TTL 标签
  • 自动输出 EC2 IP，动态生成 Ansible Inventory
  • 使用 Ansible Playbook 远程安装软件或部署服务
  • 支持多环境 stack（dev/staging/prod）

项目结构

├── config/ # 多环境配置 │ ├── base.yaml │ ├── vpc.yaml │ ├── firewall.yaml │ └── instances.yaml ├── iac_modules/ │ └── pulumi/ │ ├── deploy.py # Pulumi 主入口 │ ├── modules/ # VPC/SG/EC2 模块 │ ├── utils/config_loader.py │ └── requirements.txt ├── scripts/ │ ├── infra.sh # 一键部署脚本 │ └── inventory.py # 动态 Ansible inventory ├── ansible/ │ └── playbooks/ │ └── setup.yml # 应用部署 playbook

Phase 1: Implementing OIDC Login

In this first phase, we focus on implementing OpenID Connect (OIDC) login functionality for the following platforms:

Cloud Resouces

Cloud Host Name Public IP VPN IP Costs Service
ALI cn-gateway.svc.plus 8.130.10.142 10.254.0.1 --- cn-proxy.onwalk.net
HW hw-node.svc.plus 139.9.139.22 --- --- images.onwalk.net
AWS global-gateway.svc.plus 52.196.108.28 10.255.0.1 --- tky-connector.onwalk.net
global-images.onwalk.net
AWS us-gateway.svc.plus 54.183.199.99 --- --- us-connector.onwalk.net
grafana.onwalk.net
metrics.onwalk.net
AWS canada-gateway.svc.plus 35.182.201.252 --- --- ca-connector.onwalk.net
logs.onwalk.net

Key Components Overview

Component Description Tools/Technologies
1. LDP (Login Delegation Protocol) Centralized authentication and Single Sign-On (SSO) using Auth0 by Okta for various platforms. Auth0 by Okta, OIDC
2. IaC (Infrastructure as Code) Infrastructure management and provisioning using automated tools. Terraform, Pulumi
3. Monitoring Comprehensive observability and monitoring for the application, including system metrics, network, and performance. Grafana Cloud, Prometheus, DeepFlow, ClickHouse
4. Git Repository Version control and source code management for the project. GitHub
5. CI/CD (Continuous Integration/Delivery) Automated build, test, and deployment pipelines. GitHub Actions

key Components Service

Name Domain Version Deploy Docker Compose Chart CI/CD
1. Keycloak keycloak.onwalk.net 26.0 Docker Yes Yes GitHub Actions
2. Harbor images.onwalk.net 2.12 Docker Yes Yes GitHub Actions
3. ChartMuseum charts.onwalk.net 0.14.0 Docker Yes Yes GitHub Actions
4. Vault vault.onwalk.net 1.15 Docker Yes Yes GitHub Actions
5. Nginx/OSS mirrors.onwalk.net 1.21 Kubernetes Yes Yes GitHub Actions
Name Domain **Version Deploy Docker Compose** Chart CI/CD
5. OpenIPA freeipa.onwalk.net 4.10 Kubernetes, Docker, BareMetal Yes Yes GitHub Actions
1. PostgreSQL db.onwalk.net 16.0 Kubernetes, Docker Yes Yes GitHub Actions
8. Prometheus monitoring.onwalk.net 2.35 Kubernetes, Docker Yes Yes GitHub Actions
9. Grafana monitoring.onwalk.net 8.4 Kubernetes, Docker Yes Yes GitHub Actions
10. Consul consul.onwalk.net 1.12 Kubernetes, Docker Yes Yes GitHub Actions
12. Jenkins jenkins.onwalk.net 2.319 Kubernetes, Docker Yes Yes GitHub Actions
13. GitLab gitlab.onwalk.net 15.5 Kubernetes, Docker Yes Yes GitHub Actions
14. MinIO minio.onwalk.net 2023.2.0 Kubernetes, Docker Yes Yes GitHub Actions

1. LDP (Login Delegation Protocol) - Auth0 by Okta

  • Set up Auth0 by Okta as the identity provider to enable OpenID Connect (OIDC) login for multiple platforms:
    • AWS, GCP, Azure, GitHub, Grafana Cloud
  • OIDC allows secure Single Sign-On (SSO) across all these platforms.
  • For more details, refer to Platform-Specific OIDC Setup Docs.

2. IaC (Infrastructure as Code) - Terraform / Pulumi

  • Infrastructure for AWS, GCP, and Azure is provisioned using Terraform and Pulumi scripts.
  • These scripts allow easy and reproducible deployment and management of cloud resources.
  • See the iac/ folder for the setup files.

3. Monitoring - Grafana Cloud / Prometheus / DeepFlow / ClickHouse

  • Monitoring stack includes:
    • Prometheus for metrics collection.
    • DeepFlow for network and system observability.
    • ClickHouse for storing and querying large amounts of observability data.
    • Grafana Cloud for visualizing all collected metrics and logs.
  • Configuration files for monitoring tools can be found in the monitoring/ folder.

4. Git Repository - GitHub

  • All project code, infrastructure configurations, and documentation are managed within this GitHub repository.
  • GitHub also integrates with GitHub Actions for CI/CD.

5. CI/CD - GitHub Actions

  • Automated CI/CD pipeline is set up using GitHub Actions to ensure continuous integration and deployment.
  • Pipelines handle code testing, building, and multi-cloud deployments for platforms like AWS, GCP, and Azure.
  • YAML workflow files for GitHub Actions can be found in the .github/workflows/ directory.

For detailed instructions on configuring each platform, see:

TODO

  • Set up Auth0 by Okta as the identity provider for OIDC authentication.
  • Configure OIDC login for AWS.
  • Configure OIDC login for GCP.
  • Configure OIDC login for Azure.
  • Configure OIDC login for GitHub.
  • Configure OIDC login for Grafana Cloud.
  • Test and validate login workflows across all platforms.

Documentation

For more detailed information, please refer to the documentation available in two languages:

Getting Started

git submodule add --force https://github.com/svc-design/ansible.git git submodule add --force https://github.com/svc-design/iac_modules.git git submodule init git submodule update

New Ideas

CI Code Stages

CD Deploy Stages

  • IAC
  • ArgoCD
  • Ansible Playbook

Follow the links above to the documentation in your preferred language to get started with using this reference architecture.

Contributing

We welcome contributions to this project. If you have suggestions, improvements, or find any issues, feel free to submit a pull request.

License

This project is released under the GPL V3 license. For more details, see the LICENSE file.