Skip to main content

VM Template for Terraform

1. 🧭 Scope Legend

Use these markers to distinguish one-time template work from settings that must be unique for each Terraform clone.

1.COMMON · CONFIGURE ONCEShared keys, security rules, package baseline, and operating model that are configured once.
2.SOURCE TEMPLATE BUILDWork performed on the temporary source VM before it is sealed and converted to a Proxmox template.
3.CHANGE PER CLONEValues that must be unique per VM, especially VM name, VM ID, MAC address, DHCP reservation, and IP address.
4.VERIFY ON EVERY CLONEValidation that proves every clone generated a new machine identity and is immediately reachable through both approved SSH keys.

2. 🎯 Objective and Guaranteed Clone Behavior

COMMON · CONFIGURE ONCESOURCE TEMPLATE BUILDCHANGE PER CLONEBuild one hardened Ubuntu 26.04 source VM, preserve both approved user public keys and passwordless sudo in the template, remove only machine-specific identity, and create clones that are immediately reachable without modifying the clone.

The reusable Proxmox template is named:

trf-tmplt-ub-26-min

The finished design guarantees the following behavior:

  • prod-deploy-01 can SSH to every clone immediately with ~/.ssh/id_ed25519_ansible.
  • sudo -n works immediately for prod-deploy-01; no clone-side sudo configuration is required.
  • Windows PuTTY can SSH to every clone immediately with manoj-homelab-admin.ppk.
  • Both keys authenticate as the existing local user acllc.
  • SSH username/password authentication is disabled before sealing the template.
  • Direct root SSH login is disabled.
  • The local acllc password remains available only for emergency Proxmox-console login.
  • Both authorized public keys remain in /home/acllc/.ssh/authorized_keys.
  • No private key is stored in the VM, template, Terraform code, Git repository, or clone.
  • Every clone generates unique SSH server host keys on first boot.
  • Every clone generates a non-empty machine ID on first boot.
  • Every clone obtains its stable IP through a router DHCP reservation tied to the permanent MAC supplied by Terraform.
  • QEMU Guest Agent and Python are already present in every clone.
  • ssh.service starts normally at boot; ssh.socket remains disabled.
  • No Proxmox Cloud-Init drive or Terraform Cloud-Init settings are used.
Template access terminology:

A converted Proxmox template is not a normally running server. The source VM is accessible before conversion, and every clone is accessible immediately after first boot. The template object itself is used as the clone source.

Success criteria:

A newly cloned VM needs no SSH, key, user, sudo, package, or service changes before either approved identity can log in.

3. 🗺️ Infrastructure and Execution Map

COMMON · CONFIGURE ONCEUse the correct machine for each command. The automation private key must remain only on prod-deploy-01.
PurposeMachine or location
Proxmox web endpointhttps://192.168.8.23:8006
Proxmox node namepve
Terraform execution hostprod-terraform-01
GitHub Actions runner and Ansible control nodeprod-deploy-01
Temporary source VMtrf-tmplt-ub-26-min
Temporary source VM MACAA:BB:CC:11:22:33
Temporary source VM IP192.168.8.254
Ubuntu administrator accountacllc
Personal Windows private keymanoj-homelab-admin.ppk
Automation private keyprod-deploy-01:~/.ssh/id_ed25519_ansible

The temporary router reservation is:

AA:BB:CC:11:22:33 → 192.168.8.254
Duplicate MAC prevention:

Never power on two VMs that use AA:BB:CC:11:22:33. The source VM must be stopped before any clone uses that MAC, and production clones should normally receive different MAC addresses.

Command execution matrix

ActionRun from
Create the automation keyprod-deploy-01
Generate the personal keyWindows PuTTYgen
Create/install Ubuntu source VMProxmox UI and source VM console
Install the automation public keyprod-deploy-01
Prepare and transfer the PuTTY public keyprod-deploy-01
Configure packages, sudo, SSH, Netplan, identity regenerationSource VM
Seal and power offSource VM through Proxmox console
Convert VM to templateProxmox UI or Proxmox node shell
Run Terraformprod-terraform-01
Run SSH and Ansible validationprod-deploy-01
Validate new host fingerprint locallyClone’s Proxmox console
Test personal loginWindows PuTTY

4. 🔑 Prepare the Automation SSH Identity

COMMON · CONFIGURE ONCECreate one dedicated Ed25519 automation identity on prod-deploy-01. Its unencrypted private key supports unattended GitHub Actions and Ansible jobs.
Run on: prod-deploy-01

The private key created in this section must never leave this machine.

Create the SSH directory:

mkdir -p ~/.ssh
chmod 700 ~/.ssh

Check whether the dedicated key already exists:

ls -l \
~/.ssh/id_ed25519_ansible \
~/.ssh/id_ed25519_ansible.pub

If the key does not exist, generate it:

ssh-keygen \
-t ed25519 \
-f ~/.ssh/id_ed25519_ansible \
-C "prod-deploy-01 Ansible automation" \
-N ""

Protect the private key:

chmod 600 ~/.ssh/id_ed25519_ansible
chmod 644 ~/.ssh/id_ed25519_ansible.pub

Verify the public key:

cat ~/.ssh/id_ed25519_ansible.pub

ssh-keygen \
-lf ~/.ssh/id_ed25519_ansible.pub

Expected comment:

prod-deploy-01 Ansible automation

5. 🪟 Prepare the Personal PuTTY Identity

COMMON · CONFIGURE ONCECreate a separate passphrase-protected Ed25519 key for interactive Windows administration. Do not reuse the automation private key.
Run on: Windows workstation

Use PuTTYgen and securely retain the passphrase-protected .ppk file.

In PuTTYgen:

  1. Select EdDSA / Ed25519.
  2. Click Generate.
  3. Move the mouse until key generation completes.
  4. Set a strong key passphrase.
  5. Set the key comment to:
manoj-windows-putty
  1. Save the private key as:
manoj-homelab-admin.ppk
  1. Copy the complete value from:
Public key for pasting into OpenSSH authorized_keys file

The public line must resemble:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... manoj-windows-putty
Private-key separation:

Only the public line is installed in the template. Never place manoj-homelab-admin.ppk on Linux, in Git, or inside the VM.

6. 💿 Download and Verify Ubuntu 26.04 LTS

SOURCE TEMPLATE BUILDUse the official Ubuntu 26.04 live-server AMD64 ISO and verify its published SHA-256 checksum.
Run on: Proxmox node shell

Use the ISO-capable storage path configured on your Proxmox node.

For default local ISO storage:

cd /var/lib/vz/template/iso

Download:

wget \
https://releases.ubuntu.com/26.04/ubuntu-26.04-live-server-amd64.iso

Verify:

echo "dec49008a71f6098d0bcfc822021f4d042d5f2db279e4d75bdd981304f1ca5d9 ubuntu-26.04-live-server-amd64.iso" |
sha256sum --check

Expected:

ubuntu-26.04-live-server-amd64.iso: OK

When the ISO lives on another storage target, upload or download it through:

Proxmox → Storage → ISO Images

7. 🧱 Create the Temporary Source VM

SOURCE TEMPLATE BUILDCreate a normal ISO-installed VM. Do not add a Cloud-Init drive.
Run on: Proxmox web UI

Confirm no other active VM uses the temporary source MAC.

General

VM ID: Any unused VM ID
Name: trf-tmplt-ub-26-min
Start at boot: No

OS

ISO: ubuntu-26.04-live-server-amd64.iso
Guest OS Type: Linux

System

Machine: Default
BIOS: SeaBIOS
SCSI Controller: VirtIO SCSI single
QEMU Agent: Enabled

Do not add a Cloud-Init drive.

Disk

Bus/Device: SCSI
Storage: local-lvm
Disk size: 32 GiB
Discard: Enabled
IO thread: Enabled

CPU

Sockets: 1
Cores: 2
Type: host

Memory

Memory: 4096 MiB

Network

Bridge: vmbr0
Model: VirtIO
MAC: AA:BB:CC:11:22:33

Start the VM.

8. 🐧 Install Ubuntu Server

SOURCE TEMPLATE BUILDInstall a conventional Ubuntu Server system with DHCP and OpenSSH.
Run on: Source VM through Proxmox console

Keep the local account password for emergency console access.

Use:

Language: English
Installation type: Ubuntu Server
Network: DHCP
Proxy: Empty unless required
Mirror: Default Ubuntu mirror
Storage: Use the entire disk

Identity:

Your name: Aspireclan Administrator
Server name: trf-tmplt-ub-26-min
Username: acllc
Password: Strong unique local-console password

SSH installer page:

Install OpenSSH server: Yes
Import SSH identity: No

Finish installation and reboot.

Remove the ISO afterward:

VM → Hardware → CD/DVD Drive → Do not use any media

Verify boot order places scsi0 before the CD/DVD device.

9. 📦 Update Ubuntu and Install the Baseline

SOURCE TEMPLATE BUILDInstall all packages needed by Terraform discovery, SSH access, Ansible, and guest management.
Run on: Source VM through Proxmox console

Run these commands as acllc.

Verify basic identity and networking:

hostnamectl --static
hostname -I
ip -br address
ip route
ip -br link

Expected source IP:

192.168.8.254

Update and install packages:

sudo apt update
sudo DEBIAN_FRONTEND=noninteractive apt full-upgrade -y

sudo apt install -y \
openssh-server \
openssh-client \
qemu-guest-agent \
python3 \
python3-apt \
sudo \
curl \
ca-certificates \
jq \
git \
vim-tiny \
net-tools

Enable QEMU Guest Agent:

sudo systemctl enable --now qemu-guest-agent

Use UTC:

sudo timedatectl set-timezone UTC

Reboot after the full upgrade:

sudo reboot

After reboot:

uname -r
systemctl is-active qemu-guest-agent
python3 --version

10. 🌐 Configure DHCP by Permanent MAC Address

SOURCE TEMPLATE BUILDCHANGE PER CLONENetplan remains generic while each clone receives a Terraform-managed MAC and matching router DHCP reservation.
Run on: Source VM through Proxmox console

The interface name is discovered rather than assumed.

Determine the primary interface:

IFACE="$(ip -o route show default | awk '{print $5; exit}')"
echo "Primary interface: $IFACE"

Back up existing Netplan files:

sudo install -d -m 700 /root/netplan-template-backup
sudo cp -a /etc/netplan/. /root/netplan-template-backup/

Replace installer-generated YAML files:

sudo rm -f /etc/netplan/*.yaml

Create the DHCP configuration:

sudo tee /etc/netplan/01-template-dhcp.yaml >/dev/null <<EOF
network:
version: 2
renderer: networkd
ethernets:
${IFACE}:
dhcp4: true
dhcp-identifier: mac
optional: true
EOF

Secure and validate it:

sudo chmod 600 /etc/netplan/01-template-dhcp.yaml
sudo netplan generate
sudo netplan try
sudo netplan apply

Verify:

hostname -I
ip route

Expected source address:

192.168.8.254
Clone addressing:

Terraform does not configure an IP inside the guest. Terraform supplies a permanent MAC; the router maps that MAC to a stable IP.

11. 🤖 Install the Automation Public Key

SOURCE TEMPLATE BUILDPlace the automation public key in the source VM so it is inherited by every clone.
Run on: prod-deploy-01

This step uses the temporary Ubuntu password once. The private key stays on prod-deploy-01.

Remove a stale source-host entry if present:

ssh-keygen -R 192.168.8.254

Install the public key:

ssh-copy-id \
-i ~/.ssh/id_ed25519_ansible.pub \
acllc@192.168.8.254

Test:

ssh \
-i ~/.ssh/id_ed25519_ansible \
-o IdentitiesOnly=yes \
-o BatchMode=yes \
acllc@192.168.8.254 \
'hostname; whoami'

Expected:

trf-tmplt-ub-26-min
acllc

12. 🔐 Install the PuTTY Public Key Reliably

SOURCE TEMPLATE BUILDUse a validated temporary public-key file. Do not use an empty shell variable or copy the `.ppk` private key.
Run on: prod-deploy-01

Paste only the one-line OpenSSH public key copied from PuTTYgen.

Create a temporary file:

nano /tmp/manoj-windows-putty.pub

The file must contain exactly one line:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... manoj-windows-putty

Validate it before copying:

wc -l /tmp/manoj-windows-putty.pub
wc -c /tmp/manoj-windows-putty.pub
ssh-keygen -lf /tmp/manoj-windows-putty.pub

Expected line count:

1

Copy the public file:

scp \
-i ~/.ssh/id_ed25519_ansible \
-o IdentitiesOnly=yes \
/tmp/manoj-windows-putty.pub \
acllc@192.168.8.254:/tmp/manoj-windows-putty.pub

Install it idempotently:

ssh \
-i ~/.ssh/id_ed25519_ansible \
-o IdentitiesOnly=yes \
acllc@192.168.8.254 \
'
set -e

install -d -m 700 "$HOME/.ssh"
touch "$HOME/.ssh/authorized_keys"

sed -i "/^[[:space:]]*$/d" "$HOME/.ssh/authorized_keys"

if grep -qxFf /tmp/manoj-windows-putty.pub "$HOME/.ssh/authorized_keys"; then
echo "PuTTY public key already exists."
else
cat /tmp/manoj-windows-putty.pub >> "$HOME/.ssh/authorized_keys"
echo "PuTTY public key added."
fi

chmod 700 "$HOME/.ssh"
chmod 600 "$HOME/.ssh/authorized_keys"
rm -f /tmp/manoj-windows-putty.pub
'

Remove the local temporary file:

rm -f /tmp/manoj-windows-putty.pub

Verify the two retained public keys without printing their full bodies:

ssh \
-i ~/.ssh/id_ed25519_ansible \
-o IdentitiesOnly=yes \
acllc@192.168.8.254 \
'awk "NF {print NR \": \" \$1 \" ... \" \$NF}" "$HOME/.ssh/authorized_keys"'

Expected:

1: ssh-ed25519 ... automation
2: ssh-ed25519 ... manoj-windows-putty

The automation key comment may display only its final word because the verification prints the last field.

Do not use the failed variable method:

Do not use read -r -p ... PUTTY_PUBLIC_KEY followed by piping the variable over SSH. An empty variable can append blank lines instead of the key.

13. 🧪 Test Both SSH Identities Before Hardening

SOURCE TEMPLATE BUILDDo not disable SSH passwords until both independent key-based access paths succeed.

Automation test

Run on: prod-deploy-01

Open a new shell so the test does not depend on an existing SSH session.

ssh \
-i ~/.ssh/id_ed25519_ansible \
-o IdentitiesOnly=yes \
-o BatchMode=yes \
acllc@192.168.8.254 \
'hostname; whoami'

PuTTY test

Run on: Windows workstation

Create a completely new PuTTY session.

Session
Host Name: 192.168.8.254
Port: 22
Connection type: SSH

Connection → Data
Auto-login username: acllc

Connection → SSH → Auth → Credentials
Private key file: manoj-homelab-admin.ppk

Save as:

trf-tmplt-ub-26-min

PuTTY must request the private-key passphrase, not the Ubuntu account password.

14. 🛡️ Configure Passwordless Sudo

SOURCE TEMPLATE BUILDRetain one passwordless sudo rule in the template so prod-deploy-01 can automate every clone immediately.
Run on: Source VM

Both approved keys authenticate as acllc, so both receive the same sudo rights.

Create the rule:

echo 'acllc ALL=(ALL:ALL) NOPASSWD: ALL' |
sudo tee /etc/sudoers.d/90-acllc-ansible >/dev/null

Secure it:

sudo chmod 440 /etc/sudoers.d/90-acllc-ansible

Validate:

sudo visudo -cf /etc/sudoers.d/90-acllc-ansible

Expected:

/etc/sudoers.d/90-acllc-ansible: parsed OK

Test from prod-deploy-01:

ssh \
-i ~/.ssh/id_ed25519_ansible \
-o IdentitiesOnly=yes \
-o BatchMode=yes \
acllc@192.168.8.254 \
'sudo -n whoami'

Expected:

root

15. 🔒 Enforce Key-Only OpenSSH Access

SOURCE TEMPLATE BUILDDisable SSH passwords and direct root login while retaining local console password recovery.
Run on: Source VM

Keep the Proxmox console open and retain one working SSH session until all new-session tests pass.

Back up the main configuration:

sudo cp -a \
/etc/ssh/sshd_config \
/etc/ssh/sshd_config.original

sudo chmod a-w /etc/ssh/sshd_config.original

Create an early policy snippet:

sudo tee /etc/ssh/sshd_config.d/00-ac-key-only.conf >/dev/null <<'EOF'
PubkeyAuthentication yes
AuthenticationMethods publickey

PasswordAuthentication no
KbdInteractiveAuthentication no
PermitEmptyPasswords no

PermitRootLogin no
AllowUsers acllc

MaxAuthTries 3
LoginGraceTime 30

X11Forwarding no
AllowAgentForwarding no

DebianBanner no
EOF

Validate syntax:

sudo sshd -t

No output means success.

Inspect effective values:

sudo sshd -T |
grep -E '^(pubkeyauthentication|authenticationmethods|passwordauthentication|kbdinteractiveauthentication|permitemptypasswords|permitrootlogin|allowusers|maxauthtries|logingracetime|x11forwarding|allowagentforwarding|debianbanner) '

Expected important values:

pubkeyauthentication yes
authenticationmethods publickey
passwordauthentication no
kbdinteractiveauthentication no
permitemptypasswords no
permitrootlogin no
allowusers acllc
maxauthtries 3
x11forwarding no
allowagentforwarding no
debianbanner no

Do not restart SSH until the service-mode configuration in the next section is applied.

16. 🚪 Force Reliable ssh.service Startup

SOURCE TEMPLATE BUILDDisable OpenSSH socket activation and enable the normal daemon service. This incorporates the clone-startup correction proven during validation.
Run on: Source VM

The expected final mode is ssh.service enabled and active, with ssh.socket disabled.

Disable socket activation:

sudo systemctl disable --now ssh.socket

Remove any obsolete socket drop-in from earlier experiments:

sudo rm -rf /etc/systemd/system/ssh.socket.d

Enable and restart the normal service:

sudo systemctl daemon-reload
sudo systemctl enable ssh.service
sudo systemctl restart ssh.service

Verify:

sudo systemctl is-enabled ssh.service
sudo systemctl is-enabled ssh.socket || true
sudo systemctl status ssh.service --no-pager -l
sudo ss -lntp | grep ':22'

Expected:

ssh.service: enabled
ssh.socket: disabled
ssh.service: active (running)
port 22: listening
Why this is required:

A clone was observed with valid regenerated host keys but ssh.service inactive because the installation was relying on ssh.socket. Explicitly disabling the socket and enabling ssh.service makes first-boot SSH availability deterministic.

Retest from a new prod-deploy-01 shell:

ssh \
-i ~/.ssh/id_ed25519_ansible \
-o IdentitiesOnly=yes \
-o BatchMode=yes \
acllc@192.168.8.254 \
'hostname; whoami; sudo -n whoami'

Retest from a completely new PuTTY session.

Confirm password SSH is rejected:

ssh \
-o BatchMode=yes \
-o PubkeyAuthentication=no \
-o KbdInteractiveAuthentication=no \
-o PreferredAuthentications=password \
acllc@192.168.8.254 \
true

Expected:

Permission denied

The local acllc password remains usable through the Proxmox console.

17. 🧬 Generate Unique SSH Host Keys on First Boot

SOURCE TEMPLATE BUILDVERIFY ON EVERY CLONERemove source server host keys only during final sealing. Every clone generates replacement keys before ssh.service starts.
Run on: Source VM

Create and validate the regeneration mechanism while the current source host keys still exist.

Create the service:

sudo tee /etc/systemd/system/regenerate-ssh-host-keys.service >/dev/null <<'EOF'
[Unit]
Description=Generate unique OpenSSH host keys when missing
Documentation=man:ssh-keygen(1)
After=local-fs.target
Before=ssh.service
ConditionPathExists=!/etc/ssh/ssh_host_ed25519_key

[Service]
Type=oneshot
ExecStart=/usr/bin/ssh-keygen -A

[Install]
WantedBy=multi-user.target
EOF

Make ssh.service wait for it:

sudo mkdir -p /etc/systemd/system/ssh.service.d

sudo tee /etc/systemd/system/ssh.service.d/10-host-key-generation.conf >/dev/null <<'EOF'
[Unit]
Wants=regenerate-ssh-host-keys.service
After=regenerate-ssh-host-keys.service
EOF

Enable and validate:

sudo systemctl daemon-reload
sudo systemctl enable regenerate-ssh-host-keys.service

sudo systemd-analyze verify \
/etc/systemd/system/regenerate-ssh-host-keys.service

sudo systemctl is-enabled regenerate-ssh-host-keys.service
sudo systemctl cat regenerate-ssh-host-keys.service
sudo systemctl cat ssh.service

Do not delete the current host keys yet.

Unsafe sequence to avoid:

Never delete /etc/ssh/ssh_host_* before the regeneration service is enabled and validated. Removing the keys too early can leave port 22 unavailable.

18. ☁️ Disable Cloud-Init

SOURCE TEMPLATE BUILDThis design uses ISO installation, router DHCP reservations, retained public keys, and Ansible—not Proxmox Cloud-Init.
Run on: Source VM

Disable Cloud-Init activity even if the package was pulled in by the installer.

if command -v cloud-init >/dev/null 2>&1; then
sudo cloud-init clean --logs --seed
sudo touch /etc/cloud/cloud-init.disabled
fi

Verify:

test -f /etc/cloud/cloud-init.disabled &&
echo "Cloud-Init disabled."

In Proxmox, confirm the VM hardware contains no:

CloudInit Drive
ide2: local-lvm:cloudinit

19. Run the Complete Pre-Seal Validation

SOURCE TEMPLATE BUILDDo not seal the VM unless every required access path and service state passes.
Run on: Source VM

Run the local validation first.

echo "===== HOSTNAME ====="
hostnamectl --static

echo
echo "===== NETWORK ====="
hostname -I
ip route

echo
echo "===== AUTHORIZED KEYS ====="
sudo -u acllc test -s /home/acllc/.ssh/authorized_keys
awk 'NF {print NR ": " $1 " ... " $NF}' \
/home/acllc/.ssh/authorized_keys

grep -q 'prod-deploy-01 Ansible automation' \
/home/acllc/.ssh/authorized_keys

grep -q 'manoj-windows-putty' \
/home/acllc/.ssh/authorized_keys

echo
echo "===== AUTHORIZED KEY PERMISSIONS ====="
stat -c '%U:%G %a %n' \
/home/acllc/.ssh \
/home/acllc/.ssh/authorized_keys

echo
echo "===== SUDOERS ====="
sudo visudo -cf /etc/sudoers.d/90-acllc-ansible

echo
echo "===== SSH CONFIGURATION ====="
sudo sshd -t

echo
echo "===== SSH STARTUP MODE ====="
sudo systemctl is-enabled ssh.service
sudo systemctl is-active ssh.service
sudo systemctl is-enabled ssh.socket || true
sudo ss -lntp | grep ':22'

echo
echo "===== HOST-KEY REGENERATION ====="
sudo systemctl is-enabled regenerate-ssh-host-keys.service

echo
echo "===== QEMU AGENT ====="
sudo systemctl is-enabled qemu-guest-agent
sudo systemctl is-active qemu-guest-agent

echo
echo "===== PYTHON ====="
python3 --version

echo
echo "===== CLOUD-INIT ====="
test -f /etc/cloud/cloud-init.disabled &&
echo "Cloud-Init disabled."

echo
echo "===== PRIVATE KEY CHECK ====="
find /home/acllc -type f \
\( -name 'id_rsa' -o -name 'id_ed25519' -o -name '*.ppk' -o -name '*.pem' \) \
-print

The private-key check must print nothing.

Run on: prod-deploy-01

Run the external unattended validation.

ssh \
-i ~/.ssh/id_ed25519_ansible \
-o IdentitiesOnly=yes \
-o BatchMode=yes \
acllc@192.168.8.254 \
'hostname; whoami; sudo -n whoami'

Expected:

trf-tmplt-ub-26-min
acllc
root
Run on: Windows PuTTY

Open one final new session using manoj-homelab-admin.ppk.

Confirm:

  • PuTTY requests the private-key passphrase.
  • It does not request the Ubuntu account password.
  • whoami returns acllc.
  • sudo -n whoami returns root.

20. 🧹 Seal the Source VM

SOURCE TEMPLATE BUILDClean machine-specific state, retain both user public keys and sudo policy, remove server host keys, and power off immediately.
Run on: Source VM through Proxmox console

Do not perform the final sealing through a new SSH connection. Once host keys are removed, the VM must power off without rebooting.

Prevent additional shell history writes:

unset HISTFILE
history -c

Run the sealing script:

sudo bash <<'EOF'
set -Eeuo pipefail

echo "Validating retained access configuration..."
test -s /home/acllc/.ssh/authorized_keys
grep -q 'prod-deploy-01 Ansible automation' /home/acllc/.ssh/authorized_keys
grep -q 'manoj-windows-putty' /home/acllc/.ssh/authorized_keys
visudo -cf /etc/sudoers.d/90-acllc-ansible
sshd -t

echo "Validating SSH startup mode..."
test "$(systemctl is-enabled ssh.service)" = "enabled"
if systemctl is-enabled ssh.socket >/dev/null 2>&1; then
echo "ERROR: ssh.socket must be disabled before sealing."
exit 1
fi

echo "Validating host-key regeneration..."
test "$(systemctl is-enabled regenerate-ssh-host-keys.service)" = "enabled"
systemd-analyze verify \
/etc/systemd/system/regenerate-ssh-host-keys.service

echo "Cleaning APT data..."
apt clean
rm -rf /var/lib/apt/lists/*

echo "Cleaning Cloud-Init state..."
if command -v cloud-init >/dev/null 2>&1; then
cloud-init clean --logs --seed || true
fi
touch /etc/cloud/cloud-init.disabled

echo "Cleaning DHCP leases..."
rm -f /var/lib/dhcp/* 2>/dev/null || true
rm -f /run/systemd/netif/leases/* 2>/dev/null || true

echo "Cleaning temporary files..."
find /tmp -mindepth 1 -delete 2>/dev/null || true
find /var/tmp -mindepth 1 -delete 2>/dev/null || true

echo "Cleaning shell histories..."
rm -f /home/acllc/.bash_history
rm -f /root/.bash_history

echo "Cleaning journal..."
journalctl --rotate || true
journalctl --vacuum-time=1s || true

echo "Resetting random seed..."
rm -f /var/lib/systemd/random-seed

echo "Resetting machine identity..."
truncate -s 0 /etc/machine-id
rm -f /var/lib/dbus/machine-id
ln -s /etc/machine-id /var/lib/dbus/machine-id

echo "Removing source VM SSH server host keys..."
rm -f /etc/ssh/ssh_host_*

echo "Reconfirming retained user public keys..."
test -s /home/acllc/.ssh/authorized_keys
grep -q 'prod-deploy-01 Ansible automation' /home/acllc/.ssh/authorized_keys
grep -q 'manoj-windows-putty' /home/acllc/.ssh/authorized_keys

sync

echo "Template cleanup complete. Powering off..."
systemctl poweroff
EOF
After the script starts:

Do not interrupt it. Do not reboot the source VM after host-key removal. Wait until Proxmox reports the VM as stopped.

21. 📐 Convert the Source VM to a Proxmox Template

SOURCE TEMPLATE BUILDConvert only after the sealed source VM is fully stopped.
Run on: Proxmox web UI or Proxmox node shell

Verify the hardware and service assumptions before conversion.

Final checks:

Status: Stopped
Name: trf-tmplt-ub-26-min
CD/DVD: No media
Cloud-Init drive: None
QEMU Guest Agent: Enabled

Convert through the UI:

Select VM → More → Convert to template

Or from the Proxmox node shell:

qm template <SOURCE_VMID>

Do not start the converted template.

22. 🧮 Plan a Unique Terraform Clone

CHANGE PER CLONEEnter clone-specific values. Placeholder examples are not treated as actual values.
Router DHCP reservation:
<<UNIQUE_MAC>> → <<RESERVED_IP>>
Generated Terraform resource:
resource "proxmox_vm_qemu" "clone" {
  name        = "<<CLONE_NAME>>"
  vmid        = <<CLONE_VMID>>
  target_node = "<<PROXMOX_NODE>>"

  clone      = "trf-tmplt-ub-26-min"
  full_clone = true

  agent        = 1
  agent_timeout = 180
  skip_ipv6    = true

  boot   = "order=scsi0"
  scsihw = "virtio-scsi-single"

  cpu {
    type    = "host"
    cores   = <<CPU_CORES>>
    sockets = 1
  }

  memory = <<MEMORY_MIB>>

  disk {
    type     = "disk"
    slot     = "scsi0"
    size     = "<<DISK_SIZE_GB>>G"
    storage  = "<<STORAGE>>"
    discard  = true
    iothread = true
  }

  network {
    id      = 0
    model   = "virtio"
    bridge  = "vmbr0"
    macaddr = "<<UNIQUE_MAC>>"
  }
}
Generated SSH validation command from prod-deploy-01:
ssh \
  -i ~/.ssh/id_ed25519_ansible \
  -o IdentitiesOnly=yes \
  -o BatchMode=yes \
  acllc@<<RESERVED_IP>> \
  'hostname; whoami; sudo -n whoami'
MAC and IP uniqueness:

Before terraform apply, create the router reservation for the exact Terraform MAC and confirm no active VM uses that MAC or reserved IP.

23. 📁 Create the Terraform Working Files

COMMON · CONFIGURE ONCECHANGE PER CLONERun Terraform from prod-terraform-01. Keep provider credentials in environment variables rather than source files.
Run on: prod-terraform-01

Create or use a dedicated Terraform working directory.

mkdir -p ~/terraform/proxmox-vms
cd ~/terraform/proxmox-vms

versions.tf

terraform {
required_version = ">= 1.6.0"

required_providers {
proxmox = {
source = "Telmate/proxmox"
version = "3.0.2-rc03"
}
}
}
Provider pinning:

Keep the exact provider version that you validate in your environment. Review release notes before changing this pin.

provider.tf

provider "proxmox" {
pm_api_url = var.proxmox_api_url
pm_tls_insecure = var.proxmox_tls_insecure
}

The provider reads the API token from:

PM_API_TOKEN_ID
PM_API_TOKEN_SECRET

variables.tf

variable "proxmox_api_url" {
description = "Proxmox API endpoint including /api2/json."
type = string
}

variable "proxmox_tls_insecure" {
description = "Set true only when the Proxmox endpoint uses an untrusted lab certificate."
type = bool
default = true
}

variable "target_node" {
description = "Proxmox node that will host the clone."
type = string
}

variable "template_name" {
description = "Existing Proxmox template name."
type = string
default = "trf-tmplt-ub-26-min"
}

variable "clone_name" {
description = "Unique Proxmox VM name."
type = string
}

variable "clone_vmid" {
description = "Unique Proxmox VM ID."
type = number
}

variable "clone_mac" {
description = "Permanent unique MAC matching a router DHCP reservation."
type = string

validation {
condition = can(regex("^([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}$", var.clone_mac))
error_message = "clone_mac must use the format AA:BB:CC:DD:EE:FF."
}
}

variable "bridge" {
description = "Proxmox network bridge."
type = string
default = "vmbr0"
}

variable "storage" {
description = "Proxmox storage used for the cloned disk."
type = string
}

variable "cpu_cores" {
description = "Number of clone CPU cores."
type = number
default = 4
}

variable "memory_mib" {
description = "Clone memory in MiB."
type = number
default = 8192
}

variable "disk_size_gib" {
description = "Clone disk size in GiB. Do not set smaller than the source template disk."
type = number
default = 32
}

main.tf

resource "proxmox_vm_qemu" "clone" {
name = var.clone_name
vmid = var.clone_vmid
target_node = var.target_node

clone = var.template_name
full_clone = true

agent = 1
agent_timeout = 180
skip_ipv6 = true

boot = "order=scsi0"
scsihw = "virtio-scsi-single"

cpu {
type = "host"
cores = var.cpu_cores
sockets = 1
}

memory = var.memory_mib

disk {
type = "disk"
slot = "scsi0"
size = "${var.disk_size_gib}G"
storage = var.storage
discard = true
iothread = true
}

network {
id = 0
model = "virtio"
bridge = var.bridge
macaddr = var.clone_mac
}
}

outputs.tf

output "clone_name" {
value = proxmox_vm_qemu.clone.name
}

output "clone_vmid" {
value = proxmox_vm_qemu.clone.vmid
}

output "guest_ipv4" {
value = try(proxmox_vm_qemu.clone.default_ipv4_address, null)
}

terraform.tfvars

Example only:

proxmox_api_url = "https://192.168.8.23:8006/api2/json"
proxmox_tls_insecure = true

target_node = "pve"
template_name = "trf-tmplt-ub-26-min"

clone_name = "dev-web-99"
clone_vmid = 299
clone_mac = "AA:BB:CC:11:22:44"

bridge = "vmbr0"
storage = "local-lvm"

cpu_cores = 4
memory_mib = 8192
disk_size_gib = 32

Keep these Cloud-Init settings absent:

os_type = "cloud-init"
ciuser
cipassword
sshkeys
ipconfig0
cicustom
cloudinit disk block

.gitignore

.terraform/
*.tfstate
*.tfstate.*
.terraform.lock.hcl.backup
crash.log
*.tfvars
*.auto.tfvars

Commit .terraform.lock.hcl after successful initialization. Do not commit credentials or real .tfvars files containing sensitive values.

24. 🔏 Set Terraform Provider Credentials Safely

COMMON · CONFIGURE ONCEUse a Proxmox API token through environment variables. Do not hard-code the token secret.
Run on: prod-terraform-01

Use the API token already authorized for the required Proxmox VM operations.

export PM_API_TOKEN_ID='terraform-user@pve!terraform-token'
export PM_API_TOKEN_SECRET='REPLACE_WITH_THE_REAL_SECRET'

The ! in the token ID is safest inside single quotes.

Verify the API URL in terraform.tfvars:

https://192.168.8.23:8006/api2/json

Do not print or commit the token secret.

25. 📡 Create the Router DHCP Reservation

CHANGE PER CLONEMap the exact Terraform MAC to the desired stable IP before starting the clone.

Example:

AA:BB:CC:11:22:44 → 192.168.8.99

Confirm:

  • The MAC is unique.
  • The IP is unique.
  • The IP is within the intended network.
  • The reservation is saved and active.
  • No powered-on VM currently uses the same MAC.
  • Firewall rules permit prod-deploy-01 and the Windows workstation to reach TCP 22 on the clone.

26. 🚀 Create the Clone with Terraform

CHANGE PER CLONEInitialize, format, validate, plan, and apply from prod-terraform-01.
Run on: prod-terraform-01

The source template remains stopped. Terraform creates and starts the clone.

cd ~/terraform/proxmox-vms

terraform init
terraform fmt -recursive
terraform validate
terraform plan -out=tfplan
terraform apply tfplan

Verify in Proxmox:

VM name: matches clone_name
VM ID: matches clone_vmid
Template: trf-tmplt-ub-26-min
MAC: matches clone_mac
QEMU Agent: enabled
Cloud-Init: absent
Power state: running

27. Wait for First-Boot Identity and SSH

VERIFY ON EVERY CLONEAllow systemd to populate machine-id, generate new SSH host keys, start ssh.service, and obtain the reserved DHCP address.
Run on: prod-deploy-01

Replace the example IP with the clone’s reserved address.

CLONE_IP="192.168.8.99"

for attempt in $(seq 1 60); do
if nc -z -w 3 "$CLONE_IP" 22; then
echo "SSH is available at $CLONE_IP."
break
fi

echo "Waiting for SSH: attempt $attempt of 60"
sleep 5
done

If the loop finishes without success, use the clone’s Proxmox console and the troubleshooting section below.

28. 🧾 Verify the Clone’s New SSH Fingerprint

VERIFY ON EVERY CLONEA new fingerprint is expected because the source host keys were deliberately removed before conversion.
Run on: Clone through Proxmox console

Obtain the authoritative local fingerprint.

sudo ssh-keygen \
-lf /etc/ssh/ssh_host_ed25519_key.pub

Also verify the key files:

sudo ls -la /etc/ssh/ssh_host_*
Run on: prod-deploy-01

Remove a stale client-side entry only when that IP was previously assigned to another VM.

ssh-keygen -R 192.168.8.99

Connect interactively:

ssh \
-i ~/.ssh/id_ed25519_ansible \
-o IdentitiesOnly=yes \
acllc@192.168.8.99

Compare the displayed Ed25519 fingerprint with the Proxmox-console fingerprint before answering yes.

Client-side acceptance is not clone modification:

Removing an old known_hosts entry or accepting a newly verified host fingerprint changes only the SSH client. No configuration change is required inside the clone.

29. 🤖 Verify Immediate Automation Access and Sudo

VERIFY ON EVERY CLONEProve that the inherited automation public key and sudoers rule work without touching the clone.
Run on: prod-deploy-01

Run this after accepting the clone’s host fingerprint.

ssh \
-i ~/.ssh/id_ed25519_ansible \
-o IdentitiesOnly=yes \
-o BatchMode=yes \
acllc@192.168.8.99 \
'hostname; whoami; sudo -n whoami'

Expected initially:

trf-tmplt-ub-26-min
acllc
root

The inherited source hostname is expected until Ansible assigns the final clone hostname. No clone-side SSH or sudo setup is needed.

30. 🪟 Verify Immediate PuTTY Access

VERIFY ON EVERY CLONEProve that the inherited personal public key works on the untouched clone.
Run on: Windows workstation

Create or copy a PuTTY session for the clone IP.

Host: 192.168.8.99
Port: 22
Username: acllc
Private key: manoj-homelab-admin.ppk

On first connection:

  1. PuTTY displays the clone’s new server fingerprint.
  2. Compare it with the fingerprint obtained from the clone’s Proxmox console.
  3. Accept it only when they match.
  4. Enter the .ppk private-key passphrase.

Expected:

  • No Ubuntu account password prompt.
  • Immediate login as acllc.
  • sudo -n whoami returns root.
whoami
sudo -n whoami

Expected:

acllc
root

31. 🧬 Verify Unique Clone Identity and Service State

VERIFY ON EVERY CLONEConfirm that the clone generated identity locally and that the corrected SSH startup mode persisted.
Run on: prod-deploy-01

Run one remote audit command.

ssh \
-i ~/.ssh/id_ed25519_ansible \
-o IdentitiesOnly=yes \
acllc@192.168.8.99 \
'
set -e

echo "===== HOSTNAME ====="
hostnamectl --static

echo
echo "===== MACHINE ID ====="
cat /etc/machine-id
test -s /etc/machine-id

echo
echo "===== SSH HOST KEYS ====="
for key in /etc/ssh/ssh_host_*_key.pub; do
ssh-keygen -lf "$key"
done

echo
echo "===== SSH STARTUP MODE ====="
systemctl is-enabled ssh.service
systemctl is-active ssh.service
systemctl is-enabled ssh.socket || true
sudo ss -lntp | grep ":22"

echo
echo "===== HOST-KEY REGENERATION SERVICE ====="
systemctl is-enabled regenerate-ssh-host-keys.service
systemctl status regenerate-ssh-host-keys.service --no-pager -l || true

echo
echo "===== QEMU AGENT ====="
systemctl is-enabled qemu-guest-agent
systemctl is-active qemu-guest-agent

echo
echo "===== AUTHORIZED KEYS ====="
awk "NF {print NR \": \" \$1 \" ... \" \$NF}" \
/home/acllc/.ssh/authorized_keys
'

Expected:

machine-id: non-empty
SSH host keys: present
ssh.service: enabled and active
ssh.socket: disabled
regenerate-ssh-host-keys.service: enabled
qemu-guest-agent: enabled and active
authorized_keys: automation + manoj-windows-putty

A successful oneshot regeneration service may show inactive (dead) after completion. That is normal when it exited successfully.

32. 🅰️ Verify Ansible Without Clone-Side Changes

VERIFY ON EVERY CLONEAnsible runs from prod-deploy-01 using the inherited public key, Python, and passwordless sudo.
Run on: prod-deploy-01

Ansible is installed on the control node, not required as a package inside the clone.

Check Ansible:

ansible --version

If missing:

sudo apt update
sudo apt install -y ansible

Run the ping:

ansible all \
-i '192.168.8.99,' \
-u acllc \
--private-key ~/.ssh/id_ed25519_ansible \
--become \
-m ansible.builtin.ping

Expected:

192.168.8.99 | SUCCESS => {
"changed": false,
"ping": "pong"
}

Ansible can now assign the final hostname and role-specific configuration.

33. 🩺 Troubleshooting First-Boot SSH

VERIFY ON EVERY CLONEUse the Proxmox console only when the clone does not become reachable. Do not copy private keys into the clone.

33.1 Confirm network identity

Run on: Clone through Proxmox console
ip -br link
ip -br address
ip route
cat /etc/netplan/01-template-dhcp.yaml

Verify the NIC MAC matches Terraform and the IP matches the router reservation.

33.2 Confirm machine ID

cat /etc/machine-id

If empty:

sudo systemd-machine-id-setup
cat /etc/machine-id

33.3 Confirm host keys

Use the correct Ed25519 filename:

sudo ls -la /etc/ssh/ssh_host_*
sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

Do not use:

/etc/ssh/ssh_host_key.pub

If all host keys are missing:

sudo ssh-keygen -A

33.4 Validate SSH configuration

sudo sshd -t

No output means success.

33.5 Confirm corrected startup mode

sudo systemctl disable --now ssh.socket
sudo systemctl enable ssh.service
sudo systemctl restart ssh.service

sudo systemctl status ssh.service --no-pager -l
sudo ss -lntp | grep ':22'

Expected:

Active: active (running)
LISTEN ... :22

33.6 Inspect first-boot regeneration logs

sudo systemctl is-enabled regenerate-ssh-host-keys.service

sudo journalctl \
-u regenerate-ssh-host-keys.service \
-b \
--no-pager

33.7 Verify retained user public keys

sudo stat -c '%U:%G %a %n' \
/home/acllc/.ssh \
/home/acllc/.ssh/authorized_keys

sudo awk 'NF {print NR ": " $1 " ... " $NF}' \
/home/acllc/.ssh/authorized_keys

Expected:

1: ssh-ed25519 ... automation
2: ssh-ed25519 ... manoj-windows-putty

33.8 Understand the common command-location mistake

The automation test must be run from:

acllc@prod-deploy-01

not from:

acllc@trf-tmplt-ub-26-min

The clone intentionally does not contain:

/home/acllc/.ssh/id_ed25519_ansible

If a prompt shows the clone hostname, run:

exit

Then confirm:

hostname

Expected on the control node:

prod-deploy-01

34. ♻️ Repeatable Clone Checklist

CHANGE PER CLONEVERIFY ON EVERY CLONEUse this compact checklist for every future VM created from the template.
  1. Choose a unique VM name.
  2. Choose a unique Proxmox VM ID.
  3. Choose a permanent unique MAC.
  4. Create a router DHCP reservation for that MAC.
  5. Confirm the reserved IP is unused.
  6. Confirm the source template remains stopped.
  7. Update clone-specific Terraform values.
  8. Run terraform plan.
  9. Review the planned MAC, VM ID, target node, storage, CPU, memory, and disk.
  10. Run terraform apply.
  11. Wait for TCP 22 from prod-deploy-01.
  12. Obtain the clone’s Ed25519 host fingerprint through the Proxmox console.
  13. Remove a stale client known_hosts entry only if the IP was previously reused.
  14. Compare and accept the new host key.
  15. Test automation login and sudo -n.
  16. Test PuTTY with manoj-homelab-admin.ppk.
  17. Verify machine ID, host keys, ssh.service, disabled ssh.socket, QEMU agent, and both authorized keys.
  18. Run Ansible ping.
  19. Let Ansible assign the final hostname and server role.
  20. Never copy a private key into the clone.

35. 🧠 Security and Design Notes

COMMON · CONFIGURE ONCEUnderstand which state is intentionally shared and which state must be unique.

Intentionally shared by all clones

Local user: acllc
Automation public key: prod-deploy-01 Ansible automation
Personal public key: manoj-windows-putty
Passwordless sudo rule: /etc/sudoers.d/90-acllc-ansible
Key-only SSH policy: /etc/ssh/sshd_config.d/00-ac-key-only.conf
Netplan DHCP behavior: dhcp-identifier: mac
QEMU Guest Agent package: installed
Python: installed
Host-key regeneration service: installed and enabled

Unique on every clone

Proxmox VM ID
VM name
Permanent Terraform MAC
Router DHCP reservation
IP address
/etc/machine-id
SSH server host private keys
SSH server host public keys and fingerprints
Final hostname assigned by Ansible

Never stored in the template or clone

~/.ssh/id_ed25519_ansible
manoj-homelab-admin.ppk
Any other private SSH key
Terraform API token secret

Access consequence of the current account model

Both approved public keys authenticate as acllc. Because acllc has NOPASSWD: ALL, both the automation identity and the personal PuTTY identity can obtain root through sudo. This is intentional for the current design.

A stricter future model could use separate ansible and personal administrator accounts with different sudo policies, but that is not required for this template.

36. 📚 Primary References

COMMON · CONFIGURE ONCEUse official upstream documentation when reviewing future changes.
Finished state:

The template is complete when Terraform can create a clone that acquires its reserved IP, generates unique identity, starts ssh.service, accepts both retained public keys, provides passwordless sudo, rejects SSH passwords, and passes Ansible ping without any clone-side preparation.