refactor(vultr-vps): split declaration / shared templates / shared scripts
- config/resources/ai-workspace-hosts.yaml: resource declaration (moved from env) - templates/: shared provider.tf, variables.tf, cloud-init.yaml + hosts.tf.j2, inventory.ini.j2 (render copies the .tf/config into the env workdir) - scripts/generate.py + provision.sh: shared composition logic, parameterized by --resources/--workdir (no longer duplicated per env) - envs/ai-workspace/: degraded to a terraform workdir (only README/.gitignore tracked; rendered artifacts + tfstate gitignored) - AGENTS.md + terraform-yaml-render-pattern skill updated to the layered layout Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Haitao Pan
parent
de7fe511d7
commit
3a8065e6f0
@ -38,7 +38,8 @@ hosts.yaml (唯一人工入口：资源描述 / CMDB 源)
|
||||
- 资源信息由 env 内 `hosts.yaml` 描述；多份资源由 Jinja2 展开为**多个命名唯一的显式块**。
|
||||
- YAML 全局段经 `terraform.auto.tfvars.json` 传给 `variables.tf`；逐实例字段由 Jinja2 进 `.tf`。
|
||||
- 机密走环境变量（如 `TF_VAR_vultr_api_key`），**禁止**写入 YAML/tfvars；公钥可入 YAML。
|
||||
- 每个 env 提供 `generate.py`，至少含 `render` 与 `inventory` 两个子命令（职责见上图）。
|
||||
- 共享 `scripts/generate.py`（`--resources`/`--workdir` 参数化）提供 `render` 与
|
||||
`inventory` 两个子命令（职责见上图）；不在每个 env 各放一份。
|
||||
- Terraform 只输出运行时才确定的事实；静态字段（os_name/plan/groups/host_vars…）由 Python 合并。
|
||||
- 渲染产物（`generated_hosts.tf`、`terraform.auto.tfvars.json`、`cmdb.json`、`inventory.ini`）
|
||||
加入 `.gitignore`，不入库。
|
||||
@ -50,22 +51,32 @@ hosts.yaml (唯一人工入口：资源描述 / CMDB 源)
|
||||
- 每个用到 provider 的子模块声明 `required_providers`（含正确 `source`）。
|
||||
- OS 用 `data "vultr_os"` 按 `os_name` 解析 `os_id`，避免硬编码漂移 ID；解析不到时允许直接给 `os_id`。
|
||||
|
||||
## Reference Layout（新 env 必备文件）
|
||||
## Reference Layout（按职责分层）
|
||||
|
||||
声明 / 可复用模板 / 组合三层分离，env 目录只保留组合逻辑：
|
||||
|
||||
```
|
||||
envs/<name>/
|
||||
hosts.yaml # 唯一人工入口：global / ssh_keys / hosts
|
||||
generate.py # render + inventory 两个子命令
|
||||
templates/
|
||||
hosts.tf.j2 # 渲染逐实例 module/data + 资源块
|
||||
inventory.ini.j2 # 渲染静态 inventory
|
||||
variables.tf # 全局变量声明（值来自 tfvars.json）
|
||||
provider.tf
|
||||
provision.sh # 一键 render -> apply -> inventory -> (可选) ansible
|
||||
.gitignore # 忽略渲染产物与 .terraform/tfstate
|
||||
# 渲染产物（不入库）：generated_hosts.tf / terraform.auto.tfvars.json / cmdb.json / inventory.ini
|
||||
<provider>-vps/
|
||||
config/resources/<name>-hosts.yaml # 声明：唯一人工入口 global / ssh_keys / hosts
|
||||
templates/ # 共享：可复用 .tf 与 Jinja2 模板
|
||||
provider.tf variables.tf cloud-init.yaml # 共享 .tf/配置（render 时拷入 workdir）
|
||||
hosts.tf.j2 inventory.ini.j2 # 渲染模板
|
||||
scripts/ # 共享：组合逻辑（不依赖具体 env）
|
||||
generate.py # render + inventory；--resources / --workdir 参数化
|
||||
provision.sh # 一键 render -> apply -> inventory -> (可选) ansible
|
||||
modules/<resource>/ # 复用的资源模块
|
||||
envs/<name>/ # 运行目录（terraform workdir）
|
||||
README.md .gitignore # 唯二入库文件；其余为渲染产物 + tfstate
|
||||
# 渲染产物（落 workdir、不入库）：provider.tf / variables.tf / cloud-init.yaml /
|
||||
# generated_hosts.tf / terraform.auto.tfvars.json / cmdb.json / inventory.ini
|
||||
```
|
||||
|
||||
> 三层共享：**声明**归 `config/resources/`、**可复用 .tf 与模板**归 `templates/`、
|
||||
> **组合逻辑**归 `scripts/`；env 退化为运行目录。`scripts/generate.py render` 把
|
||||
> `templates/` 下的 provider/variables/cloud-init 拷入 workdir、渲染出 `generated_hosts.tf`，
|
||||
> 使 workdir 成为可独立 terraform 的根模块。新增一套主机只加一个 `config/resources/*.yaml`
|
||||
> + 一个 workdir，复用同一 scripts/templates。
|
||||
|
||||
## Operator Checklist（提交前自检）
|
||||
|
||||
- `terraform fmt` 无 diff；`terraform validate` 通过。
|
||||
|
||||
@ -43,7 +43,8 @@ CMDB (cmdb.json) + Ansible inventory (inventory.ini / 动态 inventory)
|
||||
|
||||
## 2. 资源描述与变量传递（MUST）
|
||||
|
||||
- 资源信息**必须**由 env 目录下的 `hosts.yaml`（或等价 `*.yaml`）描述，作为唯一人工入口。
|
||||
- 资源信息**必须**由 `config/resources/<name>-hosts.yaml`（或等价 `*.yaml`）声明，作为唯一人工入口；
|
||||
**不**放在 env 目录里。
|
||||
- YAML 的全局段经渲染写入 `terraform.auto.tfvars.json`，**传给 `variables.tf`**；
|
||||
逐实例字段由 Jinja2 展开进生成的 `.tf`。
|
||||
- 机密（API Key、私钥等）**禁止**写入 YAML / tfvars，**必须**走环境变量
|
||||
@ -51,14 +52,18 @@ CMDB (cmdb.json) + Ansible inventory (inventory.ini / 动态 inventory)
|
||||
|
||||
## 3. 渲染器约定（MUST）
|
||||
|
||||
每个采用本范式的 env **必须**提供 `generate.py`，至少含两个子命令：
|
||||
组合逻辑**必须**收敛到共享 `scripts/generate.py`（`--resources`/`--workdir` 参数化，
|
||||
不在每个 env 各放一份），至少含两个子命令：
|
||||
|
||||
- `render`：`hosts.yaml` → `generated_hosts.tf` + `terraform.auto.tfvars.json`
|
||||
- `inventory`：`terraform output`（运行时事实）+ `hosts.yaml`（静态字段）
|
||||
- `render`：`config/resources/*.yaml` → workdir 下 `generated_hosts.tf` +
|
||||
`provider.tf`/`variables.tf`/`cloud-init.yaml`（拷自 `templates/`）+ `terraform.auto.tfvars.json`
|
||||
- `inventory`：`terraform output`（运行时事实）+ YAML（静态字段）
|
||||
→ `cmdb.json` + `inventory.ini`
|
||||
|
||||
约定：
|
||||
- Jinja2 模板放 `templates/*.j2`；标识符经 `tf_id` 过滤器净化为合法 HCL 名。
|
||||
- 共享脚本放 `scripts/`，共享 .tf 与 Jinja2 模板放 `templates/`（`provider.tf`/
|
||||
`variables.tf`/`cloud-init.yaml`/`*.j2`）；env 仅作 terraform 运行目录；
|
||||
标识符经 `tf_id` 过滤器净化为合法 HCL 名。
|
||||
- Terraform 仅输出 **运行时才确定** 的事实（如 `cmdb_runtime`：ip / instance_id /
|
||||
解析后的 os_id）。静态字段（os_name/plan/groups/host_vars 等）由 Python 从 YAML 合并。
|
||||
- `inventory.ini` 中含空格的 host_var 值**必须**加引号（`key="a b c"`）。
|
||||
|
||||
@ -1,5 +1,9 @@
|
||||
# generate.py 渲染产物（由 hosts.yaml 派生，不入库）
|
||||
# generate.py 渲染产物（由 config/resources/*.yaml 与 templates/ 派生，拷入/渲染进
|
||||
# 本运行目录，均不入库）
|
||||
generated_hosts.tf
|
||||
provider.tf
|
||||
variables.tf
|
||||
cloud-init.yaml
|
||||
terraform.auto.tfvars.json
|
||||
cmdb.json
|
||||
inventory.ini
|
||||
|
||||
@ -4,21 +4,43 @@
|
||||
|
||||
- **不使用 HCL 控制结构**（无 `for_each`/`count`/`dynamic`/`templatefile` 循环）。
|
||||
- **用 Python + Jinja2 渲染 YAML** 生成显式的 Terraform `module`/`resource`/`data` 块。
|
||||
- `hosts.yaml` 描述资源信息，全局段经 `terraform.auto.tfvars.json` 传给 `variables.tf`。
|
||||
- 资源声明放共享 `config/resources/ai-workspace-hosts.yaml`，全局段经 `terraform.auto.tfvars.json` 传给 `variables.tf`。
|
||||
|
||||
默认创建两台机器（均 **4 核 8G / 公网 IP / 不开备份**）：`ai-debian13`(Debian 13)、`ai-ubuntu2604`(Ubuntu 26.04)。
|
||||
|
||||
## 目录分层（声明 / 共享模板 / 共享脚本 / 运行目录）
|
||||
|
||||
```
|
||||
vultr-vps/
|
||||
config/resources/ai-workspace-hosts.yaml # 声明：唯一人工入口 / CMDB 源
|
||||
templates/ # 共享：provider/变量/云初始化/渲染模板
|
||||
provider.tf variables.tf cloud-init.yaml hosts.tf.j2 inventory.ini.j2
|
||||
scripts/ # 共享：组合逻辑（可被多套声明复用）
|
||||
generate.py provision.sh
|
||||
modules/{compute,iam,...} # 复用的资源模块
|
||||
envs/ai-workspace/ # 运行目录（terraform workdir）：仅 README/.gitignore
|
||||
# 其余文件为渲染产物 + tfstate，均 gitignore
|
||||
```
|
||||
|
||||
> 本 env 已退化为"运行目录"：组合逻辑提到共享 `scripts/`，可复用的 .tf 与模板提到
|
||||
> 共享 `templates/`。新增一套主机只需加一个 `config/resources/<name>-hosts.yaml`
|
||||
> 和一个 workdir，复用同一套 scripts/templates。
|
||||
|
||||
## 数据流
|
||||
|
||||
```
|
||||
hosts.yaml ──generate.py render──▶ generated_hosts.tf (逐主机显式 module 块，无 for_each)
|
||||
│ └ terraform.auto.tfvars.json ──▶ variables.tf
|
||||
│
|
||||
└──────────────▶ terraform apply ──▶ output cmdb_runtime (ip/instance_id/os_id)
|
||||
│
|
||||
hosts.yaml(静态字段) + cmdb_runtime ──generate.py inventory──▶ cmdb.json + inventory.ini
|
||||
│
|
||||
playbooks/inventory/terraform_cmdb.py (Ansible 动态 inventory 读 cmdb.json)
|
||||
config/resources/ai-workspace-hosts.yaml
|
||||
│ scripts/generate.py render --resources <yaml> --workdir <env>
|
||||
│ （读 templates/hosts.tf.j2；拷 templates/{provider,variables,cloud-init} 入 workdir）
|
||||
├─▶ workdir/generated_hosts.tf (逐主机显式 module 块，无 for_each)
|
||||
├─▶ workdir/{provider.tf, variables.tf, cloud-init.yaml} (拷自 templates/)
|
||||
└─▶ workdir/terraform.auto.tfvars.json ──▶ variables.tf
|
||||
│
|
||||
└─▶ terraform -chdir=workdir apply ──▶ output cmdb_runtime (ip/instance_id/os_id)
|
||||
│
|
||||
YAML(静态字段) + cmdb_runtime ──scripts/generate.py inventory──▶ workdir/{cmdb.json, inventory.ini}
|
||||
│
|
||||
playbooks/inventory/terraform_cmdb.py (Ansible 动态 inventory 读 cmdb.json)
|
||||
```
|
||||
|
||||
> 循环逻辑全部在 Python/Jinja2 侧：每台主机 / 每个 SSH key 都被展开成
|
||||
@ -28,24 +50,23 @@ hosts.yaml(静态字段) + cmdb_runtime ──generate.py inventory──▶ cmd
|
||||
|
||||
| 文件 | 角色 |
|
||||
|------|------|
|
||||
| `hosts.yaml` | **唯一人工入口**：global / ssh_keys / hosts 资源描述 (CMDB 源) |
|
||||
| `generate.py` | `render`：YAML→tf+tfvars；`inventory`：tf 输出+YAML→cmdb.json+inventory.ini |
|
||||
| `templates/hosts.tf.j2` | 渲染逐主机 module/data 块与 `vultr_ssh_key` 资源 |
|
||||
| `templates/inventory.ini.j2` | 渲染静态 inventory.ini |
|
||||
| `variables.tf` | 全局变量声明（值来自 tfvars.json） |
|
||||
| `provider.tf` / `cloud-init.yaml` | Provider 与云初始化 |
|
||||
| `provision.sh` | 一键：render → apply → inventory →（可选）跑 Ansible |
|
||||
| `../../config/resources/ai-workspace-hosts.yaml` | **唯一人工入口**：global / ssh_keys / hosts 资源声明 (CMDB 源) |
|
||||
| `../../templates/{hosts.tf.j2, inventory.ini.j2}` | Jinja2 渲染模板（主机 module/data 块、静态 inventory） |
|
||||
| `../../templates/{provider.tf, variables.tf, cloud-init.yaml}` | 共享 .tf/配置，render 时拷入运行目录 |
|
||||
| `../../scripts/generate.py` | `render`：YAML+模板→workdir 产物；`inventory`：tf 输出+YAML→cmdb.json+inventory.ini |
|
||||
| `../../scripts/provision.sh` | 一键：render → apply → inventory →（可选）跑 Ansible |
|
||||
| `../../../../../playbooks/inventory/terraform_cmdb.py` | Ansible 动态 inventory 脚本 |
|
||||
| 本目录 `.gitignore` / `README.md` | 运行目录里唯二入库的文件 |
|
||||
|
||||
> `generated_hosts.tf` / `terraform.auto.tfvars.json` / `cmdb.json` / `inventory.ini`
|
||||
> 均为渲染产物，已在 `.gitignore` 中忽略。
|
||||
> 运行目录里的 `generated_hosts.tf` / `provider.tf` / `variables.tf` / `cloud-init.yaml` /
|
||||
> `terraform.auto.tfvars.json` / `cmdb.json` / `inventory.ini` 均为渲染产物，已 `.gitignore` 忽略。
|
||||
|
||||
## 用法
|
||||
|
||||
```bash
|
||||
export TF_VAR_vultr_api_key=xxxxxxxx
|
||||
# 编辑 hosts.yaml（填入真实 ssh 公钥 / 调整主机），然后：
|
||||
./provision.sh # 渲染 + 创建 + 生成 inventory
|
||||
# 编辑 config/resources/ai-workspace-hosts.yaml（填真实 ssh 公钥 / 调整主机），然后：
|
||||
../../scripts/provision.sh # 渲染 + 创建 + 生成 inventory
|
||||
|
||||
# 用动态 inventory 驱动 Ansible
|
||||
cd ../../../../../playbooks
|
||||
@ -53,17 +74,20 @@ ansible ai_workspace -i inventory/terraform_cmdb.py -m ping
|
||||
ansible-playbook -i inventory/terraform_cmdb.py setup-ai-workspace-all-in-one.yml
|
||||
```
|
||||
|
||||
分步执行：
|
||||
分步执行（在 vultr-vps 根目录）：
|
||||
|
||||
```bash
|
||||
python3 generate.py render # YAML -> generated_hosts.tf + terraform.auto.tfvars.json
|
||||
terraform init && terraform apply # 建机
|
||||
python3 generate.py inventory # -> cmdb.json + inventory.ini
|
||||
python3 scripts/generate.py render # -> envs/ai-workspace/ 渲染产物
|
||||
terraform -chdir=envs/ai-workspace init && terraform -chdir=envs/ai-workspace apply
|
||||
python3 scripts/generate.py inventory # -> cmdb.json + inventory.ini
|
||||
```
|
||||
|
||||
> 多套主机：加 `config/resources/<name>-hosts.yaml` + 一个 workdir，复用同一 scripts：
|
||||
> `scripts/generate.py render --resources config/resources/<name>-hosts.yaml --workdir envs/<name>`
|
||||
|
||||
## 调整主机
|
||||
|
||||
改 `hosts.yaml` 即可增删机器或改套餐/区域/分组/host_vars，重跑 `generate.py render`。
|
||||
改 `../../config/resources/ai-workspace-hosts.yaml` 即可增删机器或改套餐/区域/分组/host_vars，重跑 `scripts/generate.py render`。
|
||||
`groups` 决定主机进入哪些 Ansible 组；`host_vars` 会进入 inventory 行与动态 inventory 的 `hostvars`。
|
||||
|
||||
> OS 通过 `os_name` 自动解析 `os_id`（避免硬编码漂移的镜像 ID）。名称查不到时用
|
||||
|
||||
@ -1,180 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""hosts.yaml -> Terraform 资源 / Ansible inventory 渲染器。
|
||||
|
||||
设计要点（满足约束）：
|
||||
- 不在 HCL 里使用 for_each/count 等控制结构；用 Python + Jinja2 把 YAML
|
||||
展开成 generated_hosts.tf 中逐个的显式 module/resource/data 块。
|
||||
- hosts.yaml 的 global 段渲染成 terraform.auto.tfvars.json，传给 variables.tf。
|
||||
- apply 后用 terraform 运行时输出 + YAML 静态字段合并出 cmdb.json，
|
||||
再渲染 inventory.ini；二者供 Ansible（含动态 inventory 脚本）消费。
|
||||
|
||||
子命令：
|
||||
render hosts.yaml -> generated_hosts.tf + terraform.auto.tfvars.json
|
||||
inventory terraform output(cmdb_runtime) + hosts.yaml -> cmdb.json + inventory.ini
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import yaml
|
||||
from jinja2 import Environment, FileSystemLoader
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
HOSTS_YAML = os.path.join(HERE, "hosts.yaml")
|
||||
TEMPLATE_DIR = os.path.join(HERE, "templates")
|
||||
GENERATED_TF = os.path.join(HERE, "generated_hosts.tf")
|
||||
TFVARS_JSON = os.path.join(HERE, "terraform.auto.tfvars.json")
|
||||
CMDB_JSON = os.path.join(HERE, "cmdb.json")
|
||||
INVENTORY_INI = os.path.join(HERE, "inventory.ini")
|
||||
|
||||
|
||||
def _tf_id(value):
|
||||
"""把任意名字转成合法的 Terraform 标识符。"""
|
||||
return re.sub(r"[^0-9a-zA-Z_]", "_", str(value))
|
||||
|
||||
|
||||
def _env():
|
||||
env = Environment(
|
||||
loader=FileSystemLoader(TEMPLATE_DIR),
|
||||
trim_blocks=True,
|
||||
lstrip_blocks=False,
|
||||
keep_trailing_newline=True,
|
||||
)
|
||||
env.filters["tf_id"] = _tf_id
|
||||
return env
|
||||
|
||||
|
||||
def load_yaml():
|
||||
with open(HOSTS_YAML, encoding="utf-8") as fh:
|
||||
return yaml.safe_load(fh) or {}
|
||||
|
||||
|
||||
def cmd_render(_args):
|
||||
data = load_yaml()
|
||||
glob = data.get("global", {}) or {}
|
||||
ssh_keys = data.get("ssh_keys", []) or []
|
||||
hosts = data.get("hosts", []) or []
|
||||
|
||||
env = _env()
|
||||
rendered = env.get_template("hosts.tf.j2").render(
|
||||
ssh_keys=ssh_keys,
|
||||
hosts=hosts,
|
||||
true=True,
|
||||
false=False,
|
||||
)
|
||||
with open(GENERATED_TF, "w", encoding="utf-8") as fh:
|
||||
fh.write(rendered)
|
||||
|
||||
tfvars = {
|
||||
"region": glob.get("region", "nrt"),
|
||||
"name_prefix": glob.get("name_prefix", "ai-workspace"),
|
||||
"user_data_file": glob.get("user_data_file", "cloud-init.yaml"),
|
||||
}
|
||||
with open(TFVARS_JSON, "w", encoding="utf-8") as fh:
|
||||
json.dump(tfvars, fh, indent=2, ensure_ascii=False)
|
||||
fh.write("\n")
|
||||
|
||||
print(f" wrote {os.path.relpath(GENERATED_TF, HERE)}")
|
||||
print(f" wrote {os.path.relpath(TFVARS_JSON, HERE)}")
|
||||
print(" next: terraform init && terraform apply")
|
||||
|
||||
|
||||
def _terraform_output(name):
|
||||
out = subprocess.check_output(
|
||||
["terraform", f"-chdir={HERE}", "output", "-json", name],
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
return json.loads(out)
|
||||
|
||||
|
||||
def cmd_inventory(_args):
|
||||
data = load_yaml()
|
||||
glob = data.get("global", {}) or {}
|
||||
hosts = data.get("hosts", []) or []
|
||||
default_region = glob.get("region", "nrt")
|
||||
|
||||
try:
|
||||
runtime = _terraform_output("cmdb_runtime")
|
||||
except (OSError, subprocess.CalledProcessError) as exc:
|
||||
msg = getattr(exc, "stderr", b"") or b""
|
||||
sys.exit(
|
||||
"无法读取 terraform 输出 cmdb_runtime（请先在本目录 terraform apply）。\n"
|
||||
+ msg.decode(errors="replace")
|
||||
)
|
||||
|
||||
cmdb = {}
|
||||
groups = {}
|
||||
for host in hosts:
|
||||
name = host["name"]
|
||||
rt = runtime.get(name, {})
|
||||
host_vars = dict(host.get("host_vars", {}) or {})
|
||||
host_vars.setdefault("os_name", host.get("os_name", ""))
|
||||
host_vars.setdefault("plan", host.get("plan", "vc2-4c-8gb"))
|
||||
host_vars.setdefault("region", host.get("region") or default_region)
|
||||
|
||||
cmdb[name] = {
|
||||
"name": name,
|
||||
"ip": rt.get("ip"),
|
||||
"instance_id": rt.get("instance_id"),
|
||||
"os_id": rt.get("os_id"),
|
||||
"os_name": host.get("os_name", ""),
|
||||
"plan": host.get("plan", "vc2-4c-8gb"),
|
||||
"region": host.get("region") or default_region,
|
||||
"ansible_user": host.get("ansible_user", "root"),
|
||||
"groups": host.get("groups", []) or [],
|
||||
"tags": host.get("tags", []) or [],
|
||||
"host_vars": host_vars,
|
||||
}
|
||||
for group in cmdb[name]["groups"] or ["ungrouped"]:
|
||||
groups.setdefault(group, []).append(name)
|
||||
|
||||
with open(CMDB_JSON, "w", encoding="utf-8") as fh:
|
||||
json.dump(cmdb, fh, indent=2, ensure_ascii=False)
|
||||
fh.write("\n")
|
||||
|
||||
# 每台主机整行在 Python 侧拼好（含带引号的 host_vars），模板里只做表达式
|
||||
# 输出，避免 Jinja2 trim_blocks 把行尾 block 标签后的换行吃掉。
|
||||
lines = {}
|
||||
for name, host in cmdb.items():
|
||||
parts = [
|
||||
name,
|
||||
f"ansible_host={host['ip']}",
|
||||
f"ansible_user={host['ansible_user']}",
|
||||
]
|
||||
for k, v in host["host_vars"].items():
|
||||
parts.append(f'{k}="{v}"')
|
||||
lines[name] = " ".join(parts)
|
||||
|
||||
env = _env()
|
||||
rendered = env.get_template("inventory.ini.j2").render(
|
||||
cmdb=cmdb,
|
||||
lines=lines,
|
||||
groups={g: sorted(m) for g, m in sorted(groups.items())},
|
||||
)
|
||||
with open(INVENTORY_INI, "w", encoding="utf-8") as fh:
|
||||
fh.write(rendered)
|
||||
|
||||
print(f" wrote {os.path.relpath(CMDB_JSON, HERE)}")
|
||||
print(f" wrote {os.path.relpath(INVENTORY_INI, HERE)}")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
sub = parser.add_subparsers(dest="cmd", required=True)
|
||||
sub.add_parser("render", help="YAML -> generated_hosts.tf + tfvars").set_defaults(
|
||||
func=cmd_render
|
||||
)
|
||||
sub.add_parser(
|
||||
"inventory", help="terraform output + YAML -> cmdb.json + inventory.ini"
|
||||
).set_defaults(func=cmd_inventory)
|
||||
|
||||
args = parser.parse_args()
|
||||
args.func(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -1,15 +0,0 @@
|
||||
terraform {
|
||||
required_version = ">= 1.5"
|
||||
|
||||
required_providers {
|
||||
vultr = {
|
||||
source = "vultr/vultr"
|
||||
version = "~> 2.19"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "vultr" {
|
||||
api_key = var.vultr_api_key
|
||||
rate_limit = 700
|
||||
}
|
||||
@ -1,46 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# 一键联动：YAML -> 渲染 TF -> apply -> 生成 CMDB/inventory ->（可选）跑 Ansible
|
||||
#
|
||||
# 用法:
|
||||
# export TF_VAR_vultr_api_key=xxxx
|
||||
# ./provision.sh # 渲染+创建+生成 inventory
|
||||
# ./provision.sh ping # 之后对 ai_workspace 组跑 ping
|
||||
# ./provision.sh playbook setup-ai-workspace-all-in-one.yml
|
||||
set -euo pipefail
|
||||
|
||||
ENV_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${ENV_DIR}/../../../../.." && pwd)"
|
||||
PLAYBOOKS_DIR="${REPO_ROOT}/playbooks"
|
||||
DYN_INV="${PLAYBOOKS_DIR}/inventory/terraform_cmdb.py"
|
||||
|
||||
echo "==> [1/4] generate.py render (hosts.yaml -> generated_hosts.tf + tfvars)"
|
||||
python3 "${ENV_DIR}/generate.py" render
|
||||
|
||||
echo "==> [2/4] terraform init & apply"
|
||||
terraform -chdir="${ENV_DIR}" init -input=false >/dev/null
|
||||
terraform -chdir="${ENV_DIR}" apply -auto-approve -input=false
|
||||
|
||||
echo "==> [3/4] generate.py inventory (terraform output + YAML -> cmdb.json + inventory.ini)"
|
||||
python3 "${ENV_DIR}/generate.py" inventory
|
||||
|
||||
echo "==> [4/4] 完成。生成文件:"
|
||||
echo " ${ENV_DIR}/cmdb.json"
|
||||
echo " ${ENV_DIR}/inventory.ini"
|
||||
|
||||
action="${1:-none}"
|
||||
case "${action}" in
|
||||
none)
|
||||
echo "可手动执行: ansible ai_workspace -i ${DYN_INV} -m ping"
|
||||
;;
|
||||
ping)
|
||||
( cd "${PLAYBOOKS_DIR}" && ansible ai_workspace -i "${DYN_INV}" -m ping )
|
||||
;;
|
||||
playbook)
|
||||
pb="${2:?用法: provision.sh playbook <playbook.yml>}"
|
||||
( cd "${PLAYBOOKS_DIR}" && ansible-playbook -i "${DYN_INV}" "${pb}" )
|
||||
;;
|
||||
*)
|
||||
echo "未知动作: ${action} (支持: none|ping|playbook)" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
216
terraform-hcl-standard/vultr-vps/scripts/generate.py
Executable file
216
terraform-hcl-standard/vultr-vps/scripts/generate.py
Executable file
@ -0,0 +1,216 @@
|
||||
#!/usr/bin/env python3
|
||||
"""共享渲染器：资源声明 (config/resources) -> Terraform 资源 / Ansible inventory。
|
||||
|
||||
分层（本脚本不依赖某个具体 env，可被多套资源声明复用）：
|
||||
- 声明: ../config/resources/<name>-hosts.yaml （--resources 覆盖）
|
||||
- 共享模板: ../templates/{provider.tf, variables.tf, cloud-init.yaml,
|
||||
hosts.tf.j2, inventory.ini.j2}
|
||||
- 运行目录: ../envs/<name>/ （--workdir 覆盖；渲染产物 + tfstate 落此，均 gitignore）
|
||||
|
||||
设计要点（满足约束）：
|
||||
- 不在 HCL 里使用 for_each/count 等控制结构；用 Python + Jinja2 把 YAML
|
||||
展开成 generated_hosts.tf 中逐个的显式 module/resource/data 块。
|
||||
- YAML 的 global 段渲染成 terraform.auto.tfvars.json，传给 variables.tf。
|
||||
- apply 后用 terraform 运行时输出 + YAML 静态字段合并出 cmdb.json，
|
||||
再渲染 inventory.ini；二者供 Ansible（含动态 inventory 脚本）消费。
|
||||
|
||||
子命令：
|
||||
render YAML + 模板 -> workdir/{generated_hosts.tf, provider.tf,
|
||||
variables.tf, cloud-init.yaml, terraform.auto.tfvars.json}
|
||||
inventory terraform output(cmdb_runtime) + YAML -> workdir/{cmdb.json, inventory.ini}
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import yaml
|
||||
from jinja2 import Environment, FileSystemLoader
|
||||
|
||||
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
# scripts/ -> vultr-vps 根
|
||||
VULTR_VPS_ROOT = os.path.abspath(os.path.join(SCRIPT_DIR, ".."))
|
||||
TEMPLATE_DIR = os.path.join(VULTR_VPS_ROOT, "templates")
|
||||
|
||||
DEFAULT_RESOURCES = os.path.join(
|
||||
VULTR_VPS_ROOT, "config", "resources", "ai-workspace-hosts.yaml"
|
||||
)
|
||||
DEFAULT_WORKDIR = os.path.join(VULTR_VPS_ROOT, "envs", "ai-workspace")
|
||||
|
||||
# render 时从 templates/ 拷入运行目录的静态文件（使 workdir 成为独立根模块）。
|
||||
COPY_INTO_WORKDIR = ["provider.tf", "variables.tf", "cloud-init.yaml"]
|
||||
|
||||
|
||||
def _tf_id(value):
|
||||
"""把任意名字转成合法的 Terraform 标识符。"""
|
||||
return re.sub(r"[^0-9a-zA-Z_]", "_", str(value))
|
||||
|
||||
|
||||
def _jinja():
|
||||
env = Environment(
|
||||
loader=FileSystemLoader(TEMPLATE_DIR),
|
||||
trim_blocks=True,
|
||||
lstrip_blocks=False,
|
||||
keep_trailing_newline=True,
|
||||
)
|
||||
env.filters["tf_id"] = _tf_id
|
||||
return env
|
||||
|
||||
|
||||
def load_yaml(path):
|
||||
with open(path, encoding="utf-8") as fh:
|
||||
return yaml.safe_load(fh) or {}
|
||||
|
||||
|
||||
def _terraform_output(workdir, name):
|
||||
out = subprocess.check_output(
|
||||
["terraform", f"-chdir={workdir}", "output", "-json", name],
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
return json.loads(out)
|
||||
|
||||
|
||||
def cmd_render(args):
|
||||
resources, workdir = args.resources, args.workdir
|
||||
os.makedirs(workdir, exist_ok=True)
|
||||
data = load_yaml(resources)
|
||||
glob = data.get("global", {}) or {}
|
||||
ssh_keys = data.get("ssh_keys", []) or []
|
||||
hosts = data.get("hosts", []) or []
|
||||
|
||||
rendered = (
|
||||
_jinja()
|
||||
.get_template("hosts.tf.j2")
|
||||
.render(ssh_keys=ssh_keys, hosts=hosts, true=True, false=False)
|
||||
)
|
||||
with open(os.path.join(workdir, "generated_hosts.tf"), "w", encoding="utf-8") as fh:
|
||||
fh.write(rendered)
|
||||
|
||||
# 共享 provider/variables/cloud-init 拷入运行目录，使 workdir 成为可独立
|
||||
# terraform 的根模块（这些是渲染产物，已在 env/.gitignore 忽略）。
|
||||
for name in COPY_INTO_WORKDIR:
|
||||
shutil.copyfile(os.path.join(TEMPLATE_DIR, name), os.path.join(workdir, name))
|
||||
|
||||
tfvars = {
|
||||
"region": glob.get("region", "nrt"),
|
||||
"name_prefix": glob.get("name_prefix", "ai-workspace"),
|
||||
"user_data_file": glob.get("user_data_file", "cloud-init.yaml"),
|
||||
}
|
||||
with open(
|
||||
os.path.join(workdir, "terraform.auto.tfvars.json"), "w", encoding="utf-8"
|
||||
) as fh:
|
||||
json.dump(tfvars, fh, indent=2, ensure_ascii=False)
|
||||
fh.write("\n")
|
||||
|
||||
print(f" resources: {os.path.relpath(resources, VULTR_VPS_ROOT)}")
|
||||
print(f" workdir: {os.path.relpath(workdir, VULTR_VPS_ROOT)}")
|
||||
print(
|
||||
f" wrote generated_hosts.tf + {', '.join(COPY_INTO_WORKDIR)}"
|
||||
" + terraform.auto.tfvars.json"
|
||||
)
|
||||
print(f" next: terraform -chdir={workdir} init && terraform -chdir={workdir} apply")
|
||||
|
||||
|
||||
def cmd_inventory(args):
|
||||
resources, workdir = args.resources, args.workdir
|
||||
data = load_yaml(resources)
|
||||
glob = data.get("global", {}) or {}
|
||||
hosts = data.get("hosts", []) or []
|
||||
default_region = glob.get("region", "nrt")
|
||||
|
||||
try:
|
||||
runtime = _terraform_output(workdir, "cmdb_runtime")
|
||||
except (OSError, subprocess.CalledProcessError) as exc:
|
||||
msg = getattr(exc, "stderr", b"") or b""
|
||||
sys.exit(
|
||||
f"无法读取 terraform 输出 cmdb_runtime（请先在 {workdir} terraform apply）。\n"
|
||||
+ msg.decode(errors="replace")
|
||||
)
|
||||
|
||||
cmdb = {}
|
||||
groups = {}
|
||||
for host in hosts:
|
||||
name = host["name"]
|
||||
rt = runtime.get(name, {})
|
||||
host_vars = dict(host.get("host_vars", {}) or {})
|
||||
host_vars.setdefault("os_name", host.get("os_name", ""))
|
||||
host_vars.setdefault("plan", host.get("plan", "vc2-4c-8gb"))
|
||||
host_vars.setdefault("region", host.get("region") or default_region)
|
||||
|
||||
cmdb[name] = {
|
||||
"name": name,
|
||||
"ip": rt.get("ip"),
|
||||
"instance_id": rt.get("instance_id"),
|
||||
"os_id": rt.get("os_id"),
|
||||
"os_name": host.get("os_name", ""),
|
||||
"plan": host.get("plan", "vc2-4c-8gb"),
|
||||
"region": host.get("region") or default_region,
|
||||
"ansible_user": host.get("ansible_user", "root"),
|
||||
"groups": host.get("groups", []) or [],
|
||||
"tags": host.get("tags", []) or [],
|
||||
"host_vars": host_vars,
|
||||
}
|
||||
for group in cmdb[name]["groups"] or ["ungrouped"]:
|
||||
groups.setdefault(group, []).append(name)
|
||||
|
||||
with open(os.path.join(workdir, "cmdb.json"), "w", encoding="utf-8") as fh:
|
||||
json.dump(cmdb, fh, indent=2, ensure_ascii=False)
|
||||
fh.write("\n")
|
||||
|
||||
# 每台主机整行在 Python 侧拼好（含带引号的 host_vars），模板里只做表达式
|
||||
# 输出，避免 Jinja2 trim_blocks 把行尾 block 标签后的换行吃掉。
|
||||
lines = {}
|
||||
for name, host in cmdb.items():
|
||||
parts = [
|
||||
name,
|
||||
f"ansible_host={host['ip']}",
|
||||
f"ansible_user={host['ansible_user']}",
|
||||
]
|
||||
for k, v in host["host_vars"].items():
|
||||
parts.append(f'{k}="{v}"')
|
||||
lines[name] = " ".join(parts)
|
||||
|
||||
rendered = (
|
||||
_jinja()
|
||||
.get_template("inventory.ini.j2")
|
||||
.render(
|
||||
cmdb=cmdb,
|
||||
lines=lines,
|
||||
groups={g: sorted(m) for g, m in sorted(groups.items())},
|
||||
)
|
||||
)
|
||||
with open(os.path.join(workdir, "inventory.ini"), "w", encoding="utf-8") as fh:
|
||||
fh.write(rendered)
|
||||
|
||||
rel = os.path.relpath(workdir, VULTR_VPS_ROOT)
|
||||
print(f" wrote {os.path.join(rel, 'cmdb.json')}")
|
||||
print(f" wrote {os.path.join(rel, 'inventory.ini')}")
|
||||
|
||||
|
||||
def _add_common(p):
|
||||
p.add_argument("--resources", default=DEFAULT_RESOURCES, help="资源声明 YAML 路径")
|
||||
p.add_argument("--workdir", default=DEFAULT_WORKDIR, help="terraform 运行目录")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
sub = parser.add_subparsers(dest="cmd", required=True)
|
||||
r = sub.add_parser("render", help="YAML+模板 -> workdir 渲染产物")
|
||||
r.set_defaults(func=cmd_render)
|
||||
_add_common(r)
|
||||
i = sub.add_parser(
|
||||
"inventory", help="terraform output + YAML -> cmdb.json + inventory.ini"
|
||||
)
|
||||
i.set_defaults(func=cmd_inventory)
|
||||
_add_common(i)
|
||||
|
||||
args = parser.parse_args()
|
||||
args.func(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
57
terraform-hcl-standard/vultr-vps/scripts/provision.sh
Executable file
57
terraform-hcl-standard/vultr-vps/scripts/provision.sh
Executable file
@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env bash
|
||||
# 共享一键联动脚本：YAML 声明 -> 渲染 TF -> apply -> CMDB/inventory ->（可选）Ansible
|
||||
#
|
||||
# 位置: vultr-vps/scripts/provision.sh （与 generate.py 同目录，可被多套资源声明复用）
|
||||
#
|
||||
# 环境变量:
|
||||
# TF_VAR_vultr_api_key 必填
|
||||
# RESOURCES 资源声明 YAML（默认 config/resources/ai-workspace-hosts.yaml）
|
||||
# WORKDIR terraform 运行目录（默认 envs/ai-workspace）
|
||||
#
|
||||
# 用法:
|
||||
# export TF_VAR_vultr_api_key=xxxx
|
||||
# ./provision.sh # 渲染+创建+生成 inventory
|
||||
# ./provision.sh ping # 之后对 ai_workspace 组跑 ping
|
||||
# ./provision.sh playbook setup-ai-workspace-all-in-one.yml
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
VULTR_VPS_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||
# vultr-vps -> terraform-hcl-standard -> iac_modules -> ai-workspace-infra
|
||||
REPO_ROOT="$(cd "${VULTR_VPS_ROOT}/../../.." && pwd)"
|
||||
PLAYBOOKS_DIR="${REPO_ROOT}/playbooks"
|
||||
DYN_INV="${PLAYBOOKS_DIR}/inventory/terraform_cmdb.py"
|
||||
|
||||
RESOURCES="${RESOURCES:-${VULTR_VPS_ROOT}/config/resources/ai-workspace-hosts.yaml}"
|
||||
WORKDIR="${WORKDIR:-${VULTR_VPS_ROOT}/envs/ai-workspace}"
|
||||
GEN=("python3" "${SCRIPT_DIR}/generate.py")
|
||||
|
||||
echo "==> [1/4] render (${RESOURCES##*/} -> ${WORKDIR})"
|
||||
"${GEN[@]}" render --resources "${RESOURCES}" --workdir "${WORKDIR}"
|
||||
|
||||
echo "==> [2/4] terraform init & apply"
|
||||
terraform -chdir="${WORKDIR}" init -input=false >/dev/null
|
||||
terraform -chdir="${WORKDIR}" apply -auto-approve -input=false
|
||||
|
||||
echo "==> [3/4] inventory (terraform output + YAML -> cmdb.json + inventory.ini)"
|
||||
"${GEN[@]}" inventory --resources "${RESOURCES}" --workdir "${WORKDIR}"
|
||||
|
||||
echo "==> [4/4] 完成。生成: ${WORKDIR}/{cmdb.json,inventory.ini}"
|
||||
|
||||
action="${1:-none}"
|
||||
case "${action}" in
|
||||
none)
|
||||
echo "可手动执行: ansible ai_workspace -i ${DYN_INV} -m ping"
|
||||
;;
|
||||
ping)
|
||||
( cd "${PLAYBOOKS_DIR}" && ansible ai_workspace -i "${DYN_INV}" -m ping )
|
||||
;;
|
||||
playbook)
|
||||
pb="${2:?用法: provision.sh playbook <playbook.yml>}"
|
||||
( cd "${PLAYBOOKS_DIR}" && ansible-playbook -i "${DYN_INV}" "${pb}" )
|
||||
;;
|
||||
*)
|
||||
echo "未知动作: ${action} (支持: none|ping|playbook)" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
@ -2,12 +2,9 @@
|
||||
# terraform.auto.tfvars.json 后传入（即 “YAML 描述资源信息 -> variables.tf”）。
|
||||
# 逐主机的资源不走变量，而是由 generate.py 用 Jinja2 渲染成 generated_hosts.tf
|
||||
# 里的显式 module/resource 块（不使用 for_each/count 等 HCL 控制结构）。
|
||||
|
||||
variable "vultr_api_key" {
|
||||
description = "Vultr API Key，请通过环境变量 TF_VAR_vultr_api_key 提供（勿写入 YAML/tfvars）"
|
||||
type = string
|
||||
sensitive = true
|
||||
}
|
||||
#
|
||||
# 注意：vultr_api_key 变量与 provider 配置由 ../../templates/provider.tf 提供，
|
||||
# generate.py render 时会把它拷入本目录；故此处不再重复声明该变量。
|
||||
|
||||
variable "region" {
|
||||
description = "默认部署区域，主机未单独指定 region 时使用"
|
||||
Loading…
Reference in New Issue
Block a user