diff --git a/skills/terraform-yaml-render-pattern/SKILL.md b/skills/terraform-yaml-render-pattern/SKILL.md index 11797748..665d03d6 100644 --- a/skills/terraform-yaml-render-pattern/SKILL.md +++ b/skills/terraform-yaml-render-pattern/SKILL.md @@ -38,7 +38,8 @@ hosts.yaml (唯一人工入口：资源描述 / CMDB 源) - 资源信息由 env 内 `hosts.yaml` 描述；多份资源由 Jinja2 展开为**多个命名唯一的显式块**。 - YAML 全局段经 `terraform.auto.tfvars.json` 传给 `variables.tf`；逐实例字段由 Jinja2 进 `.tf`。 - 机密走环境变量（如 `TF_VAR_vultr_api_key`），**禁止**写入 YAML/tfvars；公钥可入 YAML。 -- 每个 env 提供 `generate.py`，至少含 `render` 与 `inventory` 两个子命令（职责见上图）。 +- 共享 `scripts/generate.py`（`--resources`/`--workdir` 参数化）提供 `render` 与 + `inventory` 两个子命令（职责见上图）；不在每个 env 各放一份。 - Terraform 只输出运行时才确定的事实；静态字段（os_name/plan/groups/host_vars…）由 Python 合并。 - 渲染产物（`generated_hosts.tf`、`terraform.auto.tfvars.json`、`cmdb.json`、`inventory.ini`） 加入 `.gitignore`，不入库。 @@ -50,22 +51,32 @@ hosts.yaml (唯一人工入口：资源描述 / CMDB 源) - 每个用到 provider 的子模块声明 `required_providers`（含正确 `source`）。 - OS 用 `data "vultr_os"` 按 `os_name` 解析 `os_id`，避免硬编码漂移 ID；解析不到时允许直接给 `os_id`。 -## Reference Layout（新 env 必备文件） +## Reference Layout（按职责分层） + +声明 / 可复用模板 / 组合三层分离，env 目录只保留组合逻辑： ``` -envs// - hosts.yaml # 唯一人工入口：global / ssh_keys / hosts - generate.py # render + inventory 两个子命令 - templates/ - hosts.tf.j2 # 渲染逐实例 module/data + 资源块 - inventory.ini.j2 # 渲染静态 inventory - variables.tf # 全局变量声明（值来自 tfvars.json） - provider.tf - provision.sh # 一键 render -> apply -> inventory -> (可选) ansible - .gitignore # 忽略渲染产物与 .terraform/tfstate - # 渲染产物（不入库）：generated_hosts.tf / terraform.auto.tfvars.json / cmdb.json / inventory.ini +-vps/ + config/resources/-hosts.yaml # 声明：唯一人工入口 global / ssh_keys / hosts + templates/ # 共享：可复用 .tf 与 Jinja2 模板 + provider.tf variables.tf cloud-init.yaml # 共享 .tf/配置（render 时拷入 workdir） + hosts.tf.j2 inventory.ini.j2 # 渲染模板 + scripts/ # 共享：组合逻辑（不依赖具体 env） + generate.py # render + inventory；--resources / --workdir 参数化 + provision.sh # 一键 render -> apply -> inventory -> (可选) ansible + modules// # 复用的资源模块 + envs// # 运行目录（terraform workdir） + README.md .gitignore # 唯二入库文件；其余为渲染产物 + tfstate + # 渲染产物（落 workdir、不入库）：provider.tf / variables.tf / cloud-init.yaml / + # generated_hosts.tf / terraform.auto.tfvars.json / cmdb.json / inventory.ini ``` +> 三层共享：**声明**归 `config/resources/`、**可复用 .tf 与模板**归 `templates/`、 +> **组合逻辑**归 `scripts/`；env 退化为运行目录。`scripts/generate.py render` 把 +> `templates/` 下的 provider/variables/cloud-init 拷入 workdir、渲染出 `generated_hosts.tf`， +> 使 workdir 成为可独立 terraform 的根模块。新增一套主机只加一个 `config/resources/*.yaml` +> + 一个 workdir，复用同一 scripts/templates。 + ## Operator Checklist（提交前自检） - `terraform fmt` 无 diff；`terraform validate` 通过。 diff --git a/terraform-hcl-standard/AGENTS.md b/terraform-hcl-standard/AGENTS.md index bf1ec486..16d37218 100644 --- a/terraform-hcl-standard/AGENTS.md +++ b/terraform-hcl-standard/AGENTS.md @@ -43,7 +43,8 @@ CMDB (cmdb.json) + Ansible inventory (inventory.ini / 动态 inventory) ## 2. 资源描述与变量传递（MUST） -- 资源信息**必须**由 env 目录下的 `hosts.yaml`（或等价 `*.yaml`）描述，作为唯一人工入口。 +- 资源信息**必须**由 `config/resources/-hosts.yaml`（或等价 `*.yaml`）声明，作为唯一人工入口； + **不**放在 env 目录里。 - YAML 的全局段经渲染写入 `terraform.auto.tfvars.json`，**传给 `variables.tf`**； 逐实例字段由 Jinja2 展开进生成的 `.tf`。 - 机密（API Key、私钥等）**禁止**写入 YAML / tfvars，**必须**走环境变量 @@ -51,14 +52,18 @@ CMDB (cmdb.json) + Ansible inventory (inventory.ini / 动态 inventory) ## 3. 渲染器约定（MUST） -每个采用本范式的 env **必须**提供 `generate.py`，至少含两个子命令： +组合逻辑**必须**收敛到共享 `scripts/generate.py`（`--resources`/`--workdir` 参数化， +不在每个 env 各放一份），至少含两个子命令： -- `render`：`hosts.yaml` → `generated_hosts.tf` + `terraform.auto.tfvars.json` -- `inventory`：`terraform output`（运行时事实）+ `hosts.yaml`（静态字段） +- `render`：`config/resources/*.yaml` → workdir 下 `generated_hosts.tf` + + `provider.tf`/`variables.tf`/`cloud-init.yaml`（拷自 `templates/`）+ `terraform.auto.tfvars.json` +- `inventory`：`terraform output`（运行时事实）+ YAML（静态字段） → `cmdb.json` + `inventory.ini` 约定： -- Jinja2 模板放 `templates/*.j2`；标识符经 `tf_id` 过滤器净化为合法 HCL 名。 +- 共享脚本放 `scripts/`，共享 .tf 与 Jinja2 模板放 `templates/`（`provider.tf`/ + `variables.tf`/`cloud-init.yaml`/`*.j2`）；env 仅作 terraform 运行目录； + 标识符经 `tf_id` 过滤器净化为合法 HCL 名。 - Terraform 仅输出 **运行时才确定** 的事实（如 `cmdb_runtime`：ip / instance_id / 解析后的 os_id）。静态字段（os_name/plan/groups/host_vars 等）由 Python 从 YAML 合并。 - `inventory.ini` 中含空格的 host_var 值**必须**加引号（`key="a b c"`）。 diff --git a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/hosts.yaml b/terraform-hcl-standard/vultr-vps/config/resources/ai-workspace-hosts.yaml similarity index 100% rename from terraform-hcl-standard/vultr-vps/envs/ai-workspace/hosts.yaml rename to terraform-hcl-standard/vultr-vps/config/resources/ai-workspace-hosts.yaml diff --git a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/.gitignore b/terraform-hcl-standard/vultr-vps/envs/ai-workspace/.gitignore index b82392d3..970c62cc 100644 --- a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/.gitignore +++ b/terraform-hcl-standard/vultr-vps/envs/ai-workspace/.gitignore @@ -1,5 +1,9 @@ -# generate.py 渲染产物（由 hosts.yaml 派生，不入库） +# generate.py 渲染产物（由 config/resources/*.yaml 与 templates/ 派生，拷入/渲染进 +# 本运行目录，均不入库） generated_hosts.tf +provider.tf +variables.tf +cloud-init.yaml terraform.auto.tfvars.json cmdb.json inventory.ini diff --git a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/README.md b/terraform-hcl-standard/vultr-vps/envs/ai-workspace/README.md index cab7a69d..da9e7910 100644 --- a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/README.md +++ b/terraform-hcl-standard/vultr-vps/envs/ai-workspace/README.md @@ -4,21 +4,43 @@ - **不使用 HCL 控制结构**（无 `for_each`/`count`/`dynamic`/`templatefile` 循环）。 - **用 Python + Jinja2 渲染 YAML** 生成显式的 Terraform `module`/`resource`/`data` 块。 -- `hosts.yaml` 描述资源信息，全局段经 `terraform.auto.tfvars.json` 传给 `variables.tf`。 +- 资源声明放共享 `config/resources/ai-workspace-hosts.yaml`，全局段经 `terraform.auto.tfvars.json` 传给 `variables.tf`。 默认创建两台机器（均 **4 核 8G / 公网 IP / 不开备份**）：`ai-debian13`(Debian 13)、`ai-ubuntu2604`(Ubuntu 26.04)。 +## 目录分层（声明 / 共享模板 / 共享脚本 / 运行目录） + +``` +vultr-vps/ + config/resources/ai-workspace-hosts.yaml # 声明：唯一人工入口 / CMDB 源 + templates/ # 共享：provider/变量/云初始化/渲染模板 + provider.tf variables.tf cloud-init.yaml hosts.tf.j2 inventory.ini.j2 + scripts/ # 共享：组合逻辑（可被多套声明复用） + generate.py provision.sh + modules/{compute,iam,...} # 复用的资源模块 + envs/ai-workspace/ # 运行目录（terraform workdir）：仅 README/.gitignore + # 其余文件为渲染产物 + tfstate，均 gitignore +``` + +> 本 env 已退化为"运行目录"：组合逻辑提到共享 `scripts/`，可复用的 .tf 与模板提到 +> 共享 `templates/`。新增一套主机只需加一个 `config/resources/-hosts.yaml` +> 和一个 workdir，复用同一套 scripts/templates。 + ## 数据流 ``` -hosts.yaml ──generate.py render──▶ generated_hosts.tf (逐主机显式 module 块，无 for_each) - │ └ terraform.auto.tfvars.json ──▶ variables.tf - │ - └──────────────▶ terraform apply ──▶ output cmdb_runtime (ip/instance_id/os_id) - │ -hosts.yaml(静态字段) + cmdb_runtime ──generate.py inventory──▶ cmdb.json + inventory.ini - │ - playbooks/inventory/terraform_cmdb.py (Ansible 动态 inventory 读 cmdb.json) +config/resources/ai-workspace-hosts.yaml + │ scripts/generate.py render --resources --workdir + │ （读 templates/hosts.tf.j2；拷 templates/{provider,variables,cloud-init} 入 workdir） + ├─▶ workdir/generated_hosts.tf (逐主机显式 module 块，无 for_each) + ├─▶ workdir/{provider.tf, variables.tf, cloud-init.yaml} (拷自 templates/) + └─▶ workdir/terraform.auto.tfvars.json ──▶ variables.tf + │ + └─▶ terraform -chdir=workdir apply ──▶ output cmdb_runtime (ip/instance_id/os_id) + │ +YAML(静态字段) + cmdb_runtime ──scripts/generate.py inventory──▶ workdir/{cmdb.json, inventory.ini} + │ + playbooks/inventory/terraform_cmdb.py (Ansible 动态 inventory 读 cmdb.json) ``` > 循环逻辑全部在 Python/Jinja2 侧：每台主机 / 每个 SSH key 都被展开成 @@ -28,24 +50,23 @@ hosts.yaml(静态字段) + cmdb_runtime ──generate.py inventory──▶ cmd | 文件 | 角色 | |------|------| -| `hosts.yaml` | **唯一人工入口**：global / ssh_keys / hosts 资源描述 (CMDB 源) | -| `generate.py` | `render`：YAML→tf+tfvars；`inventory`：tf 输出+YAML→cmdb.json+inventory.ini | -| `templates/hosts.tf.j2` | 渲染逐主机 module/data 块与 `vultr_ssh_key` 资源 | -| `templates/inventory.ini.j2` | 渲染静态 inventory.ini | -| `variables.tf` | 全局变量声明（值来自 tfvars.json） | -| `provider.tf` / `cloud-init.yaml` | Provider 与云初始化 | -| `provision.sh` | 一键：render → apply → inventory →（可选）跑 Ansible | +| `../../config/resources/ai-workspace-hosts.yaml` | **唯一人工入口**：global / ssh_keys / hosts 资源声明 (CMDB 源) | +| `../../templates/{hosts.tf.j2, inventory.ini.j2}` | Jinja2 渲染模板（主机 module/data 块、静态 inventory） | +| `../../templates/{provider.tf, variables.tf, cloud-init.yaml}` | 共享 .tf/配置，render 时拷入运行目录 | +| `../../scripts/generate.py` | `render`：YAML+模板→workdir 产物；`inventory`：tf 输出+YAML→cmdb.json+inventory.ini | +| `../../scripts/provision.sh` | 一键：render → apply → inventory →（可选）跑 Ansible | | `../../../../../playbooks/inventory/terraform_cmdb.py` | Ansible 动态 inventory 脚本 | +| 本目录 `.gitignore` / `README.md` | 运行目录里唯二入库的文件 | -> `generated_hosts.tf` / `terraform.auto.tfvars.json` / `cmdb.json` / `inventory.ini` -> 均为渲染产物，已在 `.gitignore` 中忽略。 +> 运行目录里的 `generated_hosts.tf` / `provider.tf` / `variables.tf` / `cloud-init.yaml` / +> `terraform.auto.tfvars.json` / `cmdb.json` / `inventory.ini` 均为渲染产物，已 `.gitignore` 忽略。 ## 用法 ```bash export TF_VAR_vultr_api_key=xxxxxxxx -# 编辑 hosts.yaml（填入真实 ssh 公钥 / 调整主机），然后： -./provision.sh # 渲染 + 创建 + 生成 inventory +# 编辑 config/resources/ai-workspace-hosts.yaml（填真实 ssh 公钥 / 调整主机），然后： +../../scripts/provision.sh # 渲染 + 创建 + 生成 inventory # 用动态 inventory 驱动 Ansible cd ../../../../../playbooks @@ -53,17 +74,20 @@ ansible ai_workspace -i inventory/terraform_cmdb.py -m ping ansible-playbook -i inventory/terraform_cmdb.py setup-ai-workspace-all-in-one.yml ``` -分步执行： +分步执行（在 vultr-vps 根目录）： ```bash -python3 generate.py render # YAML -> generated_hosts.tf + terraform.auto.tfvars.json -terraform init && terraform apply # 建机 -python3 generate.py inventory # -> cmdb.json + inventory.ini +python3 scripts/generate.py render # -> envs/ai-workspace/ 渲染产物 +terraform -chdir=envs/ai-workspace init && terraform -chdir=envs/ai-workspace apply +python3 scripts/generate.py inventory # -> cmdb.json + inventory.ini ``` +> 多套主机：加 `config/resources/-hosts.yaml` + 一个 workdir，复用同一 scripts： +> `scripts/generate.py render --resources config/resources/-hosts.yaml --workdir envs/` + ## 调整主机 -改 `hosts.yaml` 即可增删机器或改套餐/区域/分组/host_vars，重跑 `generate.py render`。 +改 `../../config/resources/ai-workspace-hosts.yaml` 即可增删机器或改套餐/区域/分组/host_vars，重跑 `scripts/generate.py render`。 `groups` 决定主机进入哪些 Ansible 组；`host_vars` 会进入 inventory 行与动态 inventory 的 `hostvars`。 > OS 通过 `os_name` 自动解析 `os_id`（避免硬编码漂移的镜像 ID）。名称查不到时用 diff --git a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/generate.py b/terraform-hcl-standard/vultr-vps/envs/ai-workspace/generate.py deleted file mode 100755 index 13d47ae9..00000000 --- a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/generate.py +++ /dev/null @@ -1,180 +0,0 @@ -#!/usr/bin/env python3 -"""hosts.yaml -> Terraform 资源 / Ansible inventory 渲染器。 - -设计要点（满足约束）： - - 不在 HCL 里使用 for_each/count 等控制结构；用 Python + Jinja2 把 YAML - 展开成 generated_hosts.tf 中逐个的显式 module/resource/data 块。 - - hosts.yaml 的 global 段渲染成 terraform.auto.tfvars.json，传给 variables.tf。 - - apply 后用 terraform 运行时输出 + YAML 静态字段合并出 cmdb.json， - 再渲染 inventory.ini；二者供 Ansible（含动态 inventory 脚本）消费。 - -子命令： - render hosts.yaml -> generated_hosts.tf + terraform.auto.tfvars.json - inventory terraform output(cmdb_runtime) + hosts.yaml -> cmdb.json + inventory.ini -""" - -import argparse -import json -import os -import re -import subprocess -import sys - -import yaml -from jinja2 import Environment, FileSystemLoader - -HERE = os.path.dirname(os.path.abspath(__file__)) -HOSTS_YAML = os.path.join(HERE, "hosts.yaml") -TEMPLATE_DIR = os.path.join(HERE, "templates") -GENERATED_TF = os.path.join(HERE, "generated_hosts.tf") -TFVARS_JSON = os.path.join(HERE, "terraform.auto.tfvars.json") -CMDB_JSON = os.path.join(HERE, "cmdb.json") -INVENTORY_INI = os.path.join(HERE, "inventory.ini") - - -def _tf_id(value): - """把任意名字转成合法的 Terraform 标识符。""" - return re.sub(r"[^0-9a-zA-Z_]", "_", str(value)) - - -def _env(): - env = Environment( - loader=FileSystemLoader(TEMPLATE_DIR), - trim_blocks=True, - lstrip_blocks=False, - keep_trailing_newline=True, - ) - env.filters["tf_id"] = _tf_id - return env - - -def load_yaml(): - with open(HOSTS_YAML, encoding="utf-8") as fh: - return yaml.safe_load(fh) or {} - - -def cmd_render(_args): - data = load_yaml() - glob = data.get("global", {}) or {} - ssh_keys = data.get("ssh_keys", []) or [] - hosts = data.get("hosts", []) or [] - - env = _env() - rendered = env.get_template("hosts.tf.j2").render( - ssh_keys=ssh_keys, - hosts=hosts, - true=True, - false=False, - ) - with open(GENERATED_TF, "w", encoding="utf-8") as fh: - fh.write(rendered) - - tfvars = { - "region": glob.get("region", "nrt"), - "name_prefix": glob.get("name_prefix", "ai-workspace"), - "user_data_file": glob.get("user_data_file", "cloud-init.yaml"), - } - with open(TFVARS_JSON, "w", encoding="utf-8") as fh: - json.dump(tfvars, fh, indent=2, ensure_ascii=False) - fh.write("\n") - - print(f" wrote {os.path.relpath(GENERATED_TF, HERE)}") - print(f" wrote {os.path.relpath(TFVARS_JSON, HERE)}") - print(" next: terraform init && terraform apply") - - -def _terraform_output(name): - out = subprocess.check_output( - ["terraform", f"-chdir={HERE}", "output", "-json", name], - stderr=subprocess.PIPE, - ) - return json.loads(out) - - -def cmd_inventory(_args): - data = load_yaml() - glob = data.get("global", {}) or {} - hosts = data.get("hosts", []) or [] - default_region = glob.get("region", "nrt") - - try: - runtime = _terraform_output("cmdb_runtime") - except (OSError, subprocess.CalledProcessError) as exc: - msg = getattr(exc, "stderr", b"") or b"" - sys.exit( - "无法读取 terraform 输出 cmdb_runtime（请先在本目录 terraform apply）。\n" - + msg.decode(errors="replace") - ) - - cmdb = {} - groups = {} - for host in hosts: - name = host["name"] - rt = runtime.get(name, {}) - host_vars = dict(host.get("host_vars", {}) or {}) - host_vars.setdefault("os_name", host.get("os_name", "")) - host_vars.setdefault("plan", host.get("plan", "vc2-4c-8gb")) - host_vars.setdefault("region", host.get("region") or default_region) - - cmdb[name] = { - "name": name, - "ip": rt.get("ip"), - "instance_id": rt.get("instance_id"), - "os_id": rt.get("os_id"), - "os_name": host.get("os_name", ""), - "plan": host.get("plan", "vc2-4c-8gb"), - "region": host.get("region") or default_region, - "ansible_user": host.get("ansible_user", "root"), - "groups": host.get("groups", []) or [], - "tags": host.get("tags", []) or [], - "host_vars": host_vars, - } - for group in cmdb[name]["groups"] or ["ungrouped"]: - groups.setdefault(group, []).append(name) - - with open(CMDB_JSON, "w", encoding="utf-8") as fh: - json.dump(cmdb, fh, indent=2, ensure_ascii=False) - fh.write("\n") - - # 每台主机整行在 Python 侧拼好（含带引号的 host_vars），模板里只做表达式 - # 输出，避免 Jinja2 trim_blocks 把行尾 block 标签后的换行吃掉。 - lines = {} - for name, host in cmdb.items(): - parts = [ - name, - f"ansible_host={host['ip']}", - f"ansible_user={host['ansible_user']}", - ] - for k, v in host["host_vars"].items(): - parts.append(f'{k}="{v}"') - lines[name] = " ".join(parts) - - env = _env() - rendered = env.get_template("inventory.ini.j2").render( - cmdb=cmdb, - lines=lines, - groups={g: sorted(m) for g, m in sorted(groups.items())}, - ) - with open(INVENTORY_INI, "w", encoding="utf-8") as fh: - fh.write(rendered) - - print(f" wrote {os.path.relpath(CMDB_JSON, HERE)}") - print(f" wrote {os.path.relpath(INVENTORY_INI, HERE)}") - - -def main(): - parser = argparse.ArgumentParser(description=__doc__) - sub = parser.add_subparsers(dest="cmd", required=True) - sub.add_parser("render", help="YAML -> generated_hosts.tf + tfvars").set_defaults( - func=cmd_render - ) - sub.add_parser( - "inventory", help="terraform output + YAML -> cmdb.json + inventory.ini" - ).set_defaults(func=cmd_inventory) - - args = parser.parse_args() - args.func(args) - - -if __name__ == "__main__": - main() diff --git a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/provider.tf b/terraform-hcl-standard/vultr-vps/envs/ai-workspace/provider.tf deleted file mode 100644 index f12f5aa4..00000000 --- a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/provider.tf +++ /dev/null @@ -1,15 +0,0 @@ -terraform { - required_version = ">= 1.5" - - required_providers { - vultr = { - source = "vultr/vultr" - version = "~> 2.19" - } - } -} - -provider "vultr" { - api_key = var.vultr_api_key - rate_limit = 700 -} diff --git a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/provision.sh b/terraform-hcl-standard/vultr-vps/envs/ai-workspace/provision.sh deleted file mode 100755 index c94ad223..00000000 --- a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/provision.sh +++ /dev/null @@ -1,46 +0,0 @@ -#!/usr/bin/env bash -# 一键联动：YAML -> 渲染 TF -> apply -> 生成 CMDB/inventory ->（可选）跑 Ansible -# -# 用法: -# export TF_VAR_vultr_api_key=xxxx -# ./provision.sh # 渲染+创建+生成 inventory -# ./provision.sh ping # 之后对 ai_workspace 组跑 ping -# ./provision.sh playbook setup-ai-workspace-all-in-one.yml -set -euo pipefail - -ENV_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "${ENV_DIR}/../../../../.." && pwd)" -PLAYBOOKS_DIR="${REPO_ROOT}/playbooks" -DYN_INV="${PLAYBOOKS_DIR}/inventory/terraform_cmdb.py" - -echo "==> [1/4] generate.py render (hosts.yaml -> generated_hosts.tf + tfvars)" -python3 "${ENV_DIR}/generate.py" render - -echo "==> [2/4] terraform init & apply" -terraform -chdir="${ENV_DIR}" init -input=false >/dev/null -terraform -chdir="${ENV_DIR}" apply -auto-approve -input=false - -echo "==> [3/4] generate.py inventory (terraform output + YAML -> cmdb.json + inventory.ini)" -python3 "${ENV_DIR}/generate.py" inventory - -echo "==> [4/4] 完成。生成文件:" -echo " ${ENV_DIR}/cmdb.json" -echo " ${ENV_DIR}/inventory.ini" - -action="${1:-none}" -case "${action}" in - none) - echo "可手动执行: ansible ai_workspace -i ${DYN_INV} -m ping" - ;; - ping) - ( cd "${PLAYBOOKS_DIR}" && ansible ai_workspace -i "${DYN_INV}" -m ping ) - ;; - playbook) - pb="${2:?用法: provision.sh playbook }" - ( cd "${PLAYBOOKS_DIR}" && ansible-playbook -i "${DYN_INV}" "${pb}" ) - ;; - *) - echo "未知动作: ${action} (支持: none|ping|playbook)" >&2 - exit 1 - ;; -esac diff --git a/terraform-hcl-standard/vultr-vps/scripts/generate.py b/terraform-hcl-standard/vultr-vps/scripts/generate.py new file mode 100755 index 00000000..107373e9 --- /dev/null +++ b/terraform-hcl-standard/vultr-vps/scripts/generate.py @@ -0,0 +1,216 @@ +#!/usr/bin/env python3 +"""共享渲染器：资源声明 (config/resources) -> Terraform 资源 / Ansible inventory。 + +分层（本脚本不依赖某个具体 env，可被多套资源声明复用）： + - 声明: ../config/resources/-hosts.yaml （--resources 覆盖） + - 共享模板: ../templates/{provider.tf, variables.tf, cloud-init.yaml, + hosts.tf.j2, inventory.ini.j2} + - 运行目录: ../envs// （--workdir 覆盖；渲染产物 + tfstate 落此，均 gitignore） + +设计要点（满足约束）： + - 不在 HCL 里使用 for_each/count 等控制结构；用 Python + Jinja2 把 YAML + 展开成 generated_hosts.tf 中逐个的显式 module/resource/data 块。 + - YAML 的 global 段渲染成 terraform.auto.tfvars.json，传给 variables.tf。 + - apply 后用 terraform 运行时输出 + YAML 静态字段合并出 cmdb.json， + 再渲染 inventory.ini；二者供 Ansible（含动态 inventory 脚本）消费。 + +子命令： + render YAML + 模板 -> workdir/{generated_hosts.tf, provider.tf, + variables.tf, cloud-init.yaml, terraform.auto.tfvars.json} + inventory terraform output(cmdb_runtime) + YAML -> workdir/{cmdb.json, inventory.ini} +""" + +import argparse +import json +import os +import re +import shutil +import subprocess +import sys + +import yaml +from jinja2 import Environment, FileSystemLoader + +SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) +# scripts/ -> vultr-vps 根 +VULTR_VPS_ROOT = os.path.abspath(os.path.join(SCRIPT_DIR, "..")) +TEMPLATE_DIR = os.path.join(VULTR_VPS_ROOT, "templates") + +DEFAULT_RESOURCES = os.path.join( + VULTR_VPS_ROOT, "config", "resources", "ai-workspace-hosts.yaml" +) +DEFAULT_WORKDIR = os.path.join(VULTR_VPS_ROOT, "envs", "ai-workspace") + +# render 时从 templates/ 拷入运行目录的静态文件（使 workdir 成为独立根模块）。 +COPY_INTO_WORKDIR = ["provider.tf", "variables.tf", "cloud-init.yaml"] + + +def _tf_id(value): + """把任意名字转成合法的 Terraform 标识符。""" + return re.sub(r"[^0-9a-zA-Z_]", "_", str(value)) + + +def _jinja(): + env = Environment( + loader=FileSystemLoader(TEMPLATE_DIR), + trim_blocks=True, + lstrip_blocks=False, + keep_trailing_newline=True, + ) + env.filters["tf_id"] = _tf_id + return env + + +def load_yaml(path): + with open(path, encoding="utf-8") as fh: + return yaml.safe_load(fh) or {} + + +def _terraform_output(workdir, name): + out = subprocess.check_output( + ["terraform", f"-chdir={workdir}", "output", "-json", name], + stderr=subprocess.PIPE, + ) + return json.loads(out) + + +def cmd_render(args): + resources, workdir = args.resources, args.workdir + os.makedirs(workdir, exist_ok=True) + data = load_yaml(resources) + glob = data.get("global", {}) or {} + ssh_keys = data.get("ssh_keys", []) or [] + hosts = data.get("hosts", []) or [] + + rendered = ( + _jinja() + .get_template("hosts.tf.j2") + .render(ssh_keys=ssh_keys, hosts=hosts, true=True, false=False) + ) + with open(os.path.join(workdir, "generated_hosts.tf"), "w", encoding="utf-8") as fh: + fh.write(rendered) + + # 共享 provider/variables/cloud-init 拷入运行目录，使 workdir 成为可独立 + # terraform 的根模块（这些是渲染产物，已在 env/.gitignore 忽略）。 + for name in COPY_INTO_WORKDIR: + shutil.copyfile(os.path.join(TEMPLATE_DIR, name), os.path.join(workdir, name)) + + tfvars = { + "region": glob.get("region", "nrt"), + "name_prefix": glob.get("name_prefix", "ai-workspace"), + "user_data_file": glob.get("user_data_file", "cloud-init.yaml"), + } + with open( + os.path.join(workdir, "terraform.auto.tfvars.json"), "w", encoding="utf-8" + ) as fh: + json.dump(tfvars, fh, indent=2, ensure_ascii=False) + fh.write("\n") + + print(f" resources: {os.path.relpath(resources, VULTR_VPS_ROOT)}") + print(f" workdir: {os.path.relpath(workdir, VULTR_VPS_ROOT)}") + print( + f" wrote generated_hosts.tf + {', '.join(COPY_INTO_WORKDIR)}" + " + terraform.auto.tfvars.json" + ) + print(f" next: terraform -chdir={workdir} init && terraform -chdir={workdir} apply") + + +def cmd_inventory(args): + resources, workdir = args.resources, args.workdir + data = load_yaml(resources) + glob = data.get("global", {}) or {} + hosts = data.get("hosts", []) or [] + default_region = glob.get("region", "nrt") + + try: + runtime = _terraform_output(workdir, "cmdb_runtime") + except (OSError, subprocess.CalledProcessError) as exc: + msg = getattr(exc, "stderr", b"") or b"" + sys.exit( + f"无法读取 terraform 输出 cmdb_runtime（请先在 {workdir} terraform apply）。\n" + + msg.decode(errors="replace") + ) + + cmdb = {} + groups = {} + for host in hosts: + name = host["name"] + rt = runtime.get(name, {}) + host_vars = dict(host.get("host_vars", {}) or {}) + host_vars.setdefault("os_name", host.get("os_name", "")) + host_vars.setdefault("plan", host.get("plan", "vc2-4c-8gb")) + host_vars.setdefault("region", host.get("region") or default_region) + + cmdb[name] = { + "name": name, + "ip": rt.get("ip"), + "instance_id": rt.get("instance_id"), + "os_id": rt.get("os_id"), + "os_name": host.get("os_name", ""), + "plan": host.get("plan", "vc2-4c-8gb"), + "region": host.get("region") or default_region, + "ansible_user": host.get("ansible_user", "root"), + "groups": host.get("groups", []) or [], + "tags": host.get("tags", []) or [], + "host_vars": host_vars, + } + for group in cmdb[name]["groups"] or ["ungrouped"]: + groups.setdefault(group, []).append(name) + + with open(os.path.join(workdir, "cmdb.json"), "w", encoding="utf-8") as fh: + json.dump(cmdb, fh, indent=2, ensure_ascii=False) + fh.write("\n") + + # 每台主机整行在 Python 侧拼好（含带引号的 host_vars），模板里只做表达式 + # 输出，避免 Jinja2 trim_blocks 把行尾 block 标签后的换行吃掉。 + lines = {} + for name, host in cmdb.items(): + parts = [ + name, + f"ansible_host={host['ip']}", + f"ansible_user={host['ansible_user']}", + ] + for k, v in host["host_vars"].items(): + parts.append(f'{k}="{v}"') + lines[name] = " ".join(parts) + + rendered = ( + _jinja() + .get_template("inventory.ini.j2") + .render( + cmdb=cmdb, + lines=lines, + groups={g: sorted(m) for g, m in sorted(groups.items())}, + ) + ) + with open(os.path.join(workdir, "inventory.ini"), "w", encoding="utf-8") as fh: + fh.write(rendered) + + rel = os.path.relpath(workdir, VULTR_VPS_ROOT) + print(f" wrote {os.path.join(rel, 'cmdb.json')}") + print(f" wrote {os.path.join(rel, 'inventory.ini')}") + + +def _add_common(p): + p.add_argument("--resources", default=DEFAULT_RESOURCES, help="资源声明 YAML 路径") + p.add_argument("--workdir", default=DEFAULT_WORKDIR, help="terraform 运行目录") + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest="cmd", required=True) + r = sub.add_parser("render", help="YAML+模板 -> workdir 渲染产物") + r.set_defaults(func=cmd_render) + _add_common(r) + i = sub.add_parser( + "inventory", help="terraform output + YAML -> cmdb.json + inventory.ini" + ) + i.set_defaults(func=cmd_inventory) + _add_common(i) + + args = parser.parse_args() + args.func(args) + + +if __name__ == "__main__": + main() diff --git a/terraform-hcl-standard/vultr-vps/scripts/provision.sh b/terraform-hcl-standard/vultr-vps/scripts/provision.sh new file mode 100755 index 00000000..627cd87d --- /dev/null +++ b/terraform-hcl-standard/vultr-vps/scripts/provision.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# 共享一键联动脚本：YAML 声明 -> 渲染 TF -> apply -> CMDB/inventory ->（可选）Ansible +# +# 位置: vultr-vps/scripts/provision.sh （与 generate.py 同目录，可被多套资源声明复用） +# +# 环境变量: +# TF_VAR_vultr_api_key 必填 +# RESOURCES 资源声明 YAML（默认 config/resources/ai-workspace-hosts.yaml） +# WORKDIR terraform 运行目录（默认 envs/ai-workspace） +# +# 用法: +# export TF_VAR_vultr_api_key=xxxx +# ./provision.sh # 渲染+创建+生成 inventory +# ./provision.sh ping # 之后对 ai_workspace 组跑 ping +# ./provision.sh playbook setup-ai-workspace-all-in-one.yml +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +VULTR_VPS_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" +# vultr-vps -> terraform-hcl-standard -> iac_modules -> ai-workspace-infra +REPO_ROOT="$(cd "${VULTR_VPS_ROOT}/../../.." && pwd)" +PLAYBOOKS_DIR="${REPO_ROOT}/playbooks" +DYN_INV="${PLAYBOOKS_DIR}/inventory/terraform_cmdb.py" + +RESOURCES="${RESOURCES:-${VULTR_VPS_ROOT}/config/resources/ai-workspace-hosts.yaml}" +WORKDIR="${WORKDIR:-${VULTR_VPS_ROOT}/envs/ai-workspace}" +GEN=("python3" "${SCRIPT_DIR}/generate.py") + +echo "==> [1/4] render (${RESOURCES##*/} -> ${WORKDIR})" +"${GEN[@]}" render --resources "${RESOURCES}" --workdir "${WORKDIR}" + +echo "==> [2/4] terraform init & apply" +terraform -chdir="${WORKDIR}" init -input=false >/dev/null +terraform -chdir="${WORKDIR}" apply -auto-approve -input=false + +echo "==> [3/4] inventory (terraform output + YAML -> cmdb.json + inventory.ini)" +"${GEN[@]}" inventory --resources "${RESOURCES}" --workdir "${WORKDIR}" + +echo "==> [4/4] 完成。生成: ${WORKDIR}/{cmdb.json,inventory.ini}" + +action="${1:-none}" +case "${action}" in + none) + echo "可手动执行: ansible ai_workspace -i ${DYN_INV} -m ping" + ;; + ping) + ( cd "${PLAYBOOKS_DIR}" && ansible ai_workspace -i "${DYN_INV}" -m ping ) + ;; + playbook) + pb="${2:?用法: provision.sh playbook }" + ( cd "${PLAYBOOKS_DIR}" && ansible-playbook -i "${DYN_INV}" "${pb}" ) + ;; + *) + echo "未知动作: ${action} (支持: none|ping|playbook)" >&2 + exit 1 + ;; +esac diff --git a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/cloud-init.yaml b/terraform-hcl-standard/vultr-vps/templates/cloud-init.yaml similarity index 100% rename from terraform-hcl-standard/vultr-vps/envs/ai-workspace/cloud-init.yaml rename to terraform-hcl-standard/vultr-vps/templates/cloud-init.yaml diff --git a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/templates/hosts.tf.j2 b/terraform-hcl-standard/vultr-vps/templates/hosts.tf.j2 similarity index 100% rename from terraform-hcl-standard/vultr-vps/envs/ai-workspace/templates/hosts.tf.j2 rename to terraform-hcl-standard/vultr-vps/templates/hosts.tf.j2 diff --git a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/templates/inventory.ini.j2 b/terraform-hcl-standard/vultr-vps/templates/inventory.ini.j2 similarity index 100% rename from terraform-hcl-standard/vultr-vps/envs/ai-workspace/templates/inventory.ini.j2 rename to terraform-hcl-standard/vultr-vps/templates/inventory.ini.j2 diff --git a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/variables.tf b/terraform-hcl-standard/vultr-vps/templates/variables.tf similarity index 80% rename from terraform-hcl-standard/vultr-vps/envs/ai-workspace/variables.tf rename to terraform-hcl-standard/vultr-vps/templates/variables.tf index 3d7a30e4..27fdf7b2 100644 --- a/terraform-hcl-standard/vultr-vps/envs/ai-workspace/variables.tf +++ b/terraform-hcl-standard/vultr-vps/templates/variables.tf @@ -2,12 +2,9 @@ # terraform.auto.tfvars.json 后传入（即 “YAML 描述资源信息 -> variables.tf”）。 # 逐主机的资源不走变量，而是由 generate.py 用 Jinja2 渲染成 generated_hosts.tf # 里的显式 module/resource 块（不使用 for_each/count 等 HCL 控制结构）。 - -variable "vultr_api_key" { - description = "Vultr API Key，请通过环境变量 TF_VAR_vultr_api_key 提供（勿写入 YAML/tfvars）" - type = string - sensitive = true -} +# +# 注意：vultr_api_key 变量与 provider 配置由 ../../templates/provider.tf 提供， +# generate.py render 时会把它拷入本目录；故此处不再重复声明该变量。 variable "region" { description = "默认部署区域，主机未单独指定 region 时使用"