# Role: node_monitor

> Setup Node Monitoring Exporters and Register to Infra

| **Module**        | [NODE](https://pigsty.io/docs/node)                                                              |
|-------------------|--------------------------------------------------------------------------------------------------|
| **Docs**          | https://pigsty.io/docs/node/monitor                                                              |
| **Related Roles** | [`node`](../node), [`node_id`](../node_id), [`node_remove`](../node_remove), [`infra`](../infra) |


## Overview

The `node_monitor` role sets up monitoring components for Linux nodes:

- **node_exporter**: Collects system metrics (port 9100)
- **keepalived_exporter**: Collects VIP metrics (port 9650, when VIP enabled)
- **vector**: Log collection agent (define sinks, collect syslog)

It also registers the node to infrastructure monitoring:

- **Victoria Metrics**: Register as scrape targets
- **HAProxy**: Register to Nginx reverse proxy


## Playbooks

| Playbook                     | Description                                  |
|------------------------------|----------------------------------------------|
| [`node.yml`](../../node.yml) | Full node provisioning (includes monitoring) |


## File Structure

```
roles/node_monitor/
├── defaults/
│   └── main.yml                  # Default variables
├── files/
│   ├── node_exporter.svc         # node_exporter systemd service
│   └── keepalived_exporter.svc   # keepalived_exporter systemd service
├── meta/
│   └── main.yml                  # Role dependencies
├── tasks/
│   ├── main.yml                  # Entry point
│   └── vector.yml                # Vector configuration
└── templates/
    ├── vector.toml.j2            # Vector configuration
    └── node.yaml                 # Vector node log config
```


## Tags

### Tag Hierarchy

```
node_monitor (from node.yml)
│
├── haproxy_register               # Register HAProxy to Nginx
│
├── vip_dns                        # Register VIP DNS name
│
├── node_exporter                  # node_exporter setup
│   ├── node_exporter_config       # Generate config
│   └── node_exporter_launch       # Start service
│
├── vip_exporter                   # keepalived_exporter (when VIP enabled)
│   ├── vip_exporter_config        # Generate config
│   └── vip_exporter_launch        # Start service
│
├── node_register                  # Register to monitoring
│   └── add_metrics                # Register to Victoria Metrics
│
└── vector                         # Vector log agent
    ├── vector_config              # Generate config
    └── vector_launch              # Start service
```


## Key Variables

### node_exporter

| Variable                | Default     | Description          |
|-------------------------|-------------|----------------------|
| `node_exporter_enabled` | `true`      | Enable node_exporter |
| `node_exporter_port`    | `9100`      | Listen port          |
| `node_exporter_options` | (see below) | Extra CLI options    |

**Default Options**:
```
--no-collector.softnet --no-collector.nvme --collector.tcpstat --collector.processes
```

These defaults disable noisy/problematic collectors (softnet, nvme) while enabling
useful ones (tcpstat for TCP connection stats, processes for process metrics).

### keepalived_exporter

| Variable            | Default | Description             |
|---------------------|---------|-------------------------|
| `vip_enabled`       | `false` | Enable VIP and exporter |
| `vip_exporter_port` | `9650`  | Listen port             |

### Vector

| Variable              | Default        | Description                          |
|-----------------------|----------------|--------------------------------------|
| `vector_enabled`      | `true`         | Enable vector log agent              |
| `vector_port`         | `9598`         | Metrics port                         |
| `vector_data`         | `/data/vector` | Data directory                       |
| `vector_clean`        | `false`        | Purge data dir during init           |
| `vector_read_from`    | `beginning`    | Read from beginning or end           |
| `vector_log_endpoint` | `[ infra ]`    | Log destination (see below)          |

**Log Endpoint Configuration** (`vector_log_endpoint`):

| Value                              | Behavior                                              |
|------------------------------------|-------------------------------------------------------|
| `[ infra ]`                        | Auto-discover Victoria Logs on infra nodes (default)  |
| `[ "http://host:9428/..." ]`       | Send to explicit Victoria Logs endpoint               |
| `[]` (empty)                       | Disable log forwarding, only collect metrics          |

> **Note**: Vector uses `current_boot_only: true`, meaning only logs since the
> last system boot are collected. Historical logs are not re-ingested on restart
> to avoid duplicates.

### HAProxy

| Variable                | Default | Description          |
|-------------------------|---------|----------------------|
| `haproxy_enabled`       | `true`  | Enable HAProxy       |
| `haproxy_exporter_port` | `9101`  | HAProxy metrics port |


## Exporter Ports

| Exporter            | Default Port | Description            |
|---------------------|--------------|------------------------|
| node_exporter       | 9100         | System metrics         |
| haproxy_exporter    | 9101         | HAProxy metrics        |
| vector              | 9598         | Vector metrics         |
| keepalived_exporter | 9650         | VIP/Keepalived metrics |


## Registered Targets

### Victoria Metrics

Creates target files at `/infra/targets/node/<ip>.yml`:

```yaml
- labels: { ip: 10.10.10.11, ins: pg-test-1, host: pg-test-host, cls: pg-test }
  targets:
    - 10.10.10.11:9100    # node_exporter
    - 10.10.10.11:9101    # haproxy_exporter
    - 10.10.10.11:9598    # vector
```

### Ping Targets

Creates ping targets at `/infra/targets/ping/<ip>.yml` for ICMP monitoring.


## Log Collection Architecture

Vector collects logs from journald and forwards them to Victoria Logs:

```
┌─────────────┐     ┌─────────────┐     ┌─────────────────┐
│   journald  │ ──► │   Vector    │ ──► │ Victoria Logs   │
│  (syslog)   │     │  (agent)    │     │ (infra nodes)   │
└─────────────┘     └─────────────┘     └─────────────────┘
```

**Log Fields Collected**:

| Field     | Description                           |
|-----------|---------------------------------------|
| `_time`   | Timestamp in RFC3339 format           |
| `message` | Log message content                   |
| `app`     | Application identifier (SYSLOG_IDENTIFIER) |
| `unit`    | Systemd unit name (without .service)  |
| `pid`     | Process ID                            |
| `level`   | Syslog level (info, warn, err, etc.)  |
| `ip`      | Node IP address                       |
| `ins`     | Node instance name (nodename)         |
| `cls`     | Node cluster name (node_cluster)      |
| `job`     | Log job identifier (`syslog`)         |


## See Also

- [`node`](../node): Node provisioning
- [`node_remove`](../node_remove): Remove node components
- [`infra`](../infra): Infrastructure monitoring stack
