Building the Ultimate Home Lab: The Staged Implementation Roadmap

What starts as a simple desire to run a network-wide ad blocker often snowballs into a full-scale IT infrastructure project. Over the course of building this home lab, the environment has evolved from a repurposed headless machine into a highly optimized, secure, and fully monitored Docker ecosystem.

Building a fully containerized, secure, and monitored home lab can feel overwhelming if approached all at once. To prevent cognitive overload, this master roadmap breaks the entire journey into 5 manageable, sequential stages. Each stage builds directly upon the last, taking you from bare hardware to an enterprise-grade self-hosted ecosystem.

Interactive Navigation Map

Stage 1: Foundation & Access

Hardware prep, Ubuntu Server setup, lid-sleep fixes, and Tailscale mesh VPN.

Stage 2: Containers & Core Stacks

Docker Engine, Portainer CE, Pi-hole, Plex, and Home Assistant deployments.

Stage 3: Headless Desktop & UI

TigerVNC systemd automation, XFCE4 desktop ricing, and Nginx reverse proxies.

Stage 4: Telemetry & Alerting

Prometheus, Grafana dashboards, cAdvisor, Scaphandre power tracking, and Telegram alerts.

Stage 5: Security, SIEM & Automation

Wazuh SIEM deployment, AI-powered SOC triage agents, and Ansible Infrastructure as Code (IaC).

Master Homelab Implementation Sequence

This is the complete, chronological sequence of the home lab implementation, organized into five foundational stages, advanced expansions, and diagnostics. Use this checklist to recreate the entire architecture from scratch.

Stage 1: Foundation & Secure Access

Phase 1: Flash the base OS (Ubuntu Server LTS), enable BIOS virtualization, disable Secure Boot, bypass lid-sleep, and install Tailscale.
Phase 11: Audit network diagnostics, switch to NetworkManager (nmcli), benchmark DNS, and apply firmware patches.
Phase 17: Resolve Linux LAN driver conflicts (r8168-dkms) and fix corrupt GPG/APT repository sources.

Stage 2: Core Containerization Stacks

Phase 2: Deploy Docker Engine and Portainer CE.
Phase 3: Plan core services grouping and automation workflows.
Phase 6: Deploy Pi-hole, remap port 443, and optimize database I/O starvation.
Phase 7: Deploy Plex Media Server using CIFS/SMB or direct USB bind mounts.
Phase 13: Reclaim host file ownership (PUID/PGID) and deploy Homepage dashboard.
Phase 14: Deploy Home Assistant strictly using network_mode: host for IoT discovery.

Stage 3: Headless Desktop Environment & UI

Phase 5: Install TigerVNC and XFCE4, and run headless at 16-bit color depth.
Phase 19: Optimize latency by disabling XFCE compositor and applying Arc-Dark themes.
Phase 20: Fix VNC video tearing with Xpresent blanking protocol.
Phase 24: Resolve VNC GNOME/Polkit desktop crashes via dbus-launch.
Phase 32: Secure internal web apps behind Nginx Reverse Proxy with Basic Authentication.
Phase 44: Deploy RustDesk over Tailscale for end-to-end encrypted remote desktop.
Phase 52: Configure Ubuntu's native screen sharing (Remmina).

Stage 4: Telemetry, Observability & Alerting

Phase 4: Deploy Prometheus, Node Exporter, and cAdvisor.
Phase 8: Force PCIe deep sleep states and deploy Scaphandre CPU power telemetry.
Phase 9: Write PromQL rules and deploy Alertmanager for Telegram incident alerts.
Phase 16: Update Grafana dashboards for cgroups v2 labels.
Phase 22: Upgrade Prometheus to the modern React UI.
Phase 43: Deploy cron_agent.sh to push Telegram execution logs for background cronjobs.
Phase 45: Configure Glances web server to monitor motherboard thermal readouts.

Stage 5: Enterprise Security & SIEM

Phase 10: Execute Wazuh all-in-one installation and configure Grafana OpenSearch.
Phase 31: Bypass broken APT scripts and enable anonymous OpenSearch access.
Phase 37: Fix Wazuh Manager startup timeouts and connection loops.
Phase 38: Build Wazuh IT Hygiene dashboards in Grafana using Infinity Data Source.
Phase 39 & 46: Build Real-Time AI Fast Lane for Wazuh alerts via LiteLLM.
Phase 47: Deploy Batch Processing Lane for daily AI hygiene and ISO 27001 auditing.
Phase 50: Enable docker-listener wodle to track container lifecycles in Wazuh.

Advanced Media, Hardware & Storage

Phase 12: Deploy setup_screen.sh to configure screen auto-blanking on laptops/AIOs.
Phase 15: Configure motherboard BIOS for Power Recovery and install Triage toolkits.
Phase 18: Tame Proxmox VM insomnia with PowerTOP auto-tune.
Phase 23: Manage IP routing conflicts via ARP Cache.
Phase 26: Truncate system bloat and Docker log limits.
Phase 29 & 30: Secure Kali Linux and automate daily diagnostics (dailyscri.sh).
Phase 33, 34, 35: Self-host Navidrome and build a Tailscale Funnel Telegram Kill-Switch.
Phase 36: Resolve Proxmox VM backup errors via host-level NFS mounts.
Phase 41, 42, 48: Deploy Immich (VA-API/OpenVINO), Jellyfin CSS, and Raspberry Pi OMV.
Phase 51: Essential Sysadmin CLI Extras (Fastfetch, Flatpak, Mac HFS+).
Phase 53: Automate nodes via Ansible (Infrastructure as Code).
Phase 54: Ultimate Homelab Auto-Backup Pipeline.

Phase 1: Transforming an Old Laptop into a Server

An old laptop is arguably the best entry-level home lab hardware. It has a built-in UPS (the battery), an integrated keyboard/monitor for troubleshooting, and is highly energy-efficient.

1. Install the Base OS

Flash a USB drive with Ubuntu Server LTS and install it. Choose the minimal installation option to keep background processes to an absolute minimum.

2. Prevent Sleep on Lid Close

Modify the systemd logind configuration to keep your server running when closed:

sudo nano /etc/systemd/logind.conf

Find #HandleLidSwitch=suspend, uncomment it, and change it to ignore. Then restart the service:

sudo systemctl restart systemd-logind

3. Secure Networking via Tailscale

Instead of opening ports on your router, install Tailscale to create a secure mesh network:

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --ssh

Phase 2: Deploying the Docker Engine

Containerization isolates your applications, prevents dependency conflicts, and makes backups trivial.

1. Install Official Docker

Install the official Docker repository to ensure you receive the latest engine updates:

sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "$VERSION_CODENAME")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y

2. Deploy Portainer CE

Manage your containers visually by deploying Portainer.

docker volume create portainer_data
docker run -d -p 9443:9443 --name portainer --restart=always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v portainer_data:/data \
  portainer/portainer-ce:latest

Phase 3: Core Services & Automation Planning

With the infrastructure established, you can begin deploying production stacks using Docker Compose. This phase serves as your architectural blueprint. The exact deployment scripts and configurations for these services are covered in their respective dedicated phases below:

Home Assistant: Deploy using network_mode: host to ensure it can seamlessly discover IoT devices. (Implementation scripts detailed in Phase 14).
Nginx Reverse Proxy: Route traffic and secure internal toolkits via HTTP Basic Authentication (htpasswd). (Implementation scripts detailed in Phase 32).
Immich & PostgreSQL: Build a self-hosted photo backup solution requiring strict database configurations. (Implementation scripts detailed in Phase 48).
Wazuh SIEM: Protect your fleet by ingesting security logs over your Tailscale VPN network. (Implementation scripts detailed in Phase 10, 31, 37, 38, and 50).

Phase 4: Global Observability (Prometheus, cAdvisor, Scaphandre)

A home lab is flying blind without proper telemetry. Run Node Exporter on the host OS to track disk I/O, CPU wait times, and RAM saturation. Deploy cAdvisor to monitor Docker containers, and Scaphandre to pull direct RAPL wattage metrics.

1. Primary Prometheus Config (`/etc/prometheus/prometheus.yml`)

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['localhost:8082']
  - job_name: 'scaphandre'
    static_configs:
      - targets: ['localhost:8081']

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

2. Prometheus Alert Rules (`/etc/prometheus/alert.rules.yml`)

groups:
  - name: homelab_alerts
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."

      - alert: HighCpuLoad
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU load on {{ $labels.instance }}"
          description: "CPU load is over 90% for the last 5 minutes."

Phase 5: Configuring the Headless VNC Environment

Traditional VNC setups struggle with modern hardware-accelerated desktops. By pairing TigerVNC with XFCE4, you eliminate crashes and drastically reduce idle RAM usage.

1. The Startup Script

mkdir -p ~/.config/tigervnc
nano ~/.config/tigervnc/xstartup

Paste this optimized execution block:

#!/bin/sh
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &

export XDG_RUNTIME_DIR=/run/user/$(id -u)
xhost +local:

exec dbus-launch --exit-with-session startxfce4

Make it executable: chmod +x ~/.config/tigervnc/xstartup

2. The Systemd Automation File

Run at 16-bit color depth (-depth 16) to slash network transmission overhead:

sudo nano /etc/systemd/system/vncserver@.service

[Unit]
Description=Start TigerVNC server at startup
After=syslog.target network.target

[Service]
Type=forking
User=<YOUR_USERNAME>
WorkingDirectory=/home/<YOUR_USERNAME>
Environment=HOME=/home/<YOUR_USERNAME>

ExecStartPre=-/usr/bin/vncserver -kill :%i > /dev/null 2>&1
ExecStart=/usr/bin/vncserver -depth 16 -geometry 1920x1080 -localhost no :%i
ExecStop=/usr/bin/vncserver -kill :%i

[Install]
WantedBy=multi-user.target

Start the service: sudo systemctl enable --now vncserver@1.service

Phase 6: Pi-hole Deployment & I/O Optimization

1. The Compose File

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "80:80/tcp"
      - "8443:443/tcp" # Altered host port
    environment:
      TZ: 'Europe/London'
      WEBPASSWORD: '<YOUR_SECURE_PASSWORD>'
      FTLCONF_dns_listeningMode: 'ALL'
    volumes:
      - './etc-pihole:/etc/pihole'
      - './etc-dnsmasq.d:/etc/dnsmasq.d'
    cap_add:
      - SYS_NICE
    restart: unless-stopped

2. Fixing the "High Load Average" Database Freeze

Restrict the database lifespan to 7 days to maintain lightning-fast I/O performance:

sudo systemctl stop pihole-FTL
sudo rm /etc/pihole/pihole-FTL.db
sudo nano /etc/pihole/pihole-FTL.conf

Add MAXDBDAYS=7, then restart: sudo systemctl start pihole-FTL.

Phase 7: Plex Media Server (NAS vs USB Storage)

Plex requires host networking to broadcast discovery protocols successfully.

services:
  plex:
    image: lscr.io/linuxserver/plex:latest
    container_name: plex
    network_mode: host
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London
    volumes:
      - ./config:/config
      - nas_media:/data/nas_shared
    restart: unless-stopped

volumes:
  nas_media:
    driver: local
    driver_opts:
      type: cifs
      o: "username=${NAS_USER},password=${NAS_PASS},iocharset=utf8,vers=3.0"
      device: "//<NAS_IP_ADDRESS>/<SHARE_NAME>"

For USB Mounts: Replace the volume with - /mnt/usb_drive:/data/usb_media and ensure the host directory has chmod -R 755 permissions.

Phase 8: Hardware Power Tuning & Scaphandre

Slash idle power draw by forcing PCIe ASPM and deploying Scaphandre telemetry.

# 1. Force PCIe ASPM Sleep States
git clone https://github.com/K0mh0li0/autoaspm.git
sudo nano /etc/systemd/system/autoaspm.service

# 2. Deploy Scaphandre Exporter
wget https://github.com/hubblo-org/scaphandre/releases/download/v1.0.2/scaphandre_v1.0.2-deb12_amd64.deb
sudo dpkg -i scaphandre_v1.0.2-deb12_amd64.deb

Phase 9: Real-Time Alerts via Telegram

Configure Alertmanager (/etc/alertmanager/alertmanager.yml) to format firing and resolved events into clean HTML push notifications sent directly to your Telegram bot:

route:
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: 'telegram-bot'

receivers:
- name: 'telegram-bot'
  telegram_configs:
    - send_resolved: true
      bot_token: '<YOUR_BOT_TOKEN_ID:SECRET_STRING>'
      chat_id: -<YOUR_GROUP_CHAT_ID>
      parse_mode: 'HTML'
      message: |-
        {{ if eq .Status "firing" }} 🔥 FIRING {{ else }} ✅ RESOLVED {{ end }}: {{ .CommonLabels.alertname }}
        Host: {{ .CommonLabels.instance }}
        Details: {{ .CommonAnnotations.description }}

Restart the stack: sudo systemctl restart prometheus alertmanager.

Phase 10: Enterprise SIEM Deployment (Wazuh)

Use the official script to deploy Wazuh, and securely add the agent keys to avoid APT errors:

curl -sO https://packages.wazuh.com/4.x/wazuh-install.sh
sudo bash wazuh-install.sh -a

Phase 11: Network Diagnostics & Hardware Firmware

Manage headless Wi-Fi connections via nmcli and update your BIOS/UEFI revocation lists using fwupdmgr update to block Secure Boot vulnerabilities.

Phase 12: Hardware Auditing & Custom Power Automation

For All-in-One machines, the physical screen backlight must be disabled overnight to save electricity. Create setup_screen.sh to automatically detect the sysfs backlight interface and deploy root-level cron jobs to shut the screen off.

Phase 13: Identity Management & Homepage Dashboard

Map your Process User ID (PUID) and Process Group ID (PGID) in Docker to avoid EACCES errors.

services:
  homepage:
    image: ghcr.io/gethomepage/homepage:latest
    container_name: homepage
    ports:
      - "3001:3000"
    volumes:
      - ./homepage/config:/app/config # The correct internal path
    environment:
      - PUID=1000
      - PGID=1000

Homepage Configurations (`settings.yml` & `services.yml`)

# settings.yml
title: "Homelab Dashboard"
theme: dark
layout:
  Core Infrastructure:
    style: columns
    columns: 3

# services.yml
- Core Infrastructure:
    - Portainer:
        icon: portainer.png
        href: https://<YOUR_SERVER_IP>:9443
        description: Container Management
        container: portainer

Phase 14: Deploying Home Assistant

Home Assistant requires network_mode: host to broadcast discovery protocols to IoT devices.

services:
  homeassistant:
    container_name: homeassistant
    image: ghcr.io/home-assistant/home-assistant:stable
    volumes:
      - ./config:/config
      - /etc/localtime:/etc/localtime:ro
    network_mode: host

Phase 15: Advanced Server Triage & Automated Power Recovery

When your server slows down but standard htop shows low CPU usage, you are likely experiencing I/O Wait or Network exhaustion. Equip your terminal with modern diagnostic alternatives like btop, iotop, and iftop.

Automating AC Power Recovery (BIOS Level)

If your home experiences a power outage, your headless server will remain off when power returns unless instructed otherwise by the motherboard. You cannot configure this from Linux; it must be done in the BIOS/UEFI.

Reboot the server and tap Del or F2 to enter the BIOS (e.g., MSI Click BIOS 5).
Switch to Advanced Mode (F7).
Navigate to Settings > Advanced > Power Management Setup.
Locate Restore after AC Power Loss and change it from Power Off to Power On.

Now, the moment your smart plug or UPS restores power, the server will instantly pull up the Ubuntu bootloader without physical intervention.

Phase 16: Tuning Grafana Variables for cgroups v2

When utilizing modern Linux environments running cgroups v2, cAdvisor exports metrics slightly differently than legacy systems. If you import popular Grafana dashboards and encounter "No Data", it is because the internal PromQL queries are looking for a name= label, while modern cAdvisor maps container endpoints to the id= label (e.g., /system.slice/docker.service).

1. Fixing Dropdown Variable Queries

Navigate to Dashboard Settings > Variables and update the Classic Query expressions:

Host Variable: label_values(container_cpu_usage_seconds_total, instance)
Container Variable: label_values(container_cpu_usage_seconds_total{instance=~"$host", id=~"/system.slice/.*"}, id)

2. Updating Panel Metric Queries

# CPU Usage Query Fix
sum(rate(container_cpu_usage_seconds_total{instance=~"$host", id=~"$container"}[5m])) by (id) * 100

# Memory Working Set Query Fix
container_memory_working_set_bytes{instance=~"$host", id=~"$container"}

Phase 17: Fixing Linux LAN Drivers & APT Repositories

1. The Realtek LAN `link=no` Bug

If your Ethernet cable is plugged in, but running sudo lshw -C network shows link=no, you have likely hit a known kernel conflict with Realtek RTL8111/8168 chips. Compile the proprietary driver:

sudo apt update
sudo apt install r8168-dkms -y
sudo reboot

2. Fixing GPG `NO_PUBKEY` Errors

If apt update throws key errors, purge the corrupted lists and re-import the token:

# Example: Fixing a broken Wazuh Repo
sudo rm -f /etc/apt/sources.list.d/wazuh.list
sudo rm -rf /var/lib/apt/lists/*wazuh*
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg --import
sudo chmod 644 /usr/share/keyrings/wazuh.gpg
echo "deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://packages.wazuh.com/4.x/apt/ stable main" | sudo tee /etc/apt/sources.list.d/wazuh.list
sudo apt update

Phase 18: Taming VM Insomnia on Proxmox

If you transition to Proxmox, VMs often pull 10W-15W at "idle." Background polling forces the physical CPU to wake from deep C-states thousands of times per second.

Apply PowerTOP Auto-Tune

Because you passed the physical CPU through to the VM (qm set <VM_ID> --cpu host), the guest OS can manage its virtual power states. Install and run PowerTOP inside the guest:

sudo apt update && sudo apt install powertop -y
sudo powertop --auto-tune

(Add @reboot /usr/sbin/powertop --auto-tune to the root crontab to make this persistent).

Phase 19: XFCE "Ricing" (Aesthetics & Speed Optimization)

Out of the box, XFCE's default animations can cripple a VNC network stream. This blueprint transforms the interface into a sleek, flat workspace while slashing network latency.

1. The Speed Layer

Kill the Compositor: Open Settings > Window Manager Tweaks > Compositor and uncheck Enable display compositing.
Solid Wallpaper: Right-click the desktop, select Desktop Settings, and change the Style to None (Solid color) with a dark grey hex code.

2. The Aesthetics Layer

sudo apt update
sudo apt install arc-theme papirus-icon-theme xfce4-whiskermenu-plugin -y

Apply the new skin via the XFCE Settings menu: Set Style and Window Borders to Arc-Dark, and Icons to Papirus-Dark.

Phase 20: Handling Video Tearing in VNC

If you explicitly require video playback on a headless server, you must force XFCE to use a low-overhead sync method to prevent horizontal tearing:

# Run this in your VNC terminal
xfconf-query -c xfwm4 -p /general/vblank_mode -s xpresent

Then, go back to Window Manager Tweaks > Compositor and re-check Enable display compositing. You must also force media players to bypass hardware acceleration (e.g., set VLC output to X11 video output (XCB)).

Phase 21: The Modern Sysadmin Terminal Toolkit & Automated Triage

When your server feels sluggish but standard htop shows plenty of free CPU and RAM, you are likely suffering from silent disk I/O saturation or network bandwidth exhaustion. Upgrade your triage workflow:

btop: A gorgeous, mouse-responsive TUI with real-time graphs for CPU cores, disks, and networking.
iostat: Run iostat -xz 1 to monitor hardware saturation. If %util is near 100%, your drive is struggling.
iotop: Run sudo iotop -o to view a live list of processes actively reading and writing to disk.
iftop: Run sudo iftop to inspect active IP connections and pinpoint heavy data streams.

Building an Automated System Watchdog

We can deploy a background watchdog script to automatically flush caches during high I/O wait, clear out Docker bloat, and alert your phone.

sudo nano /root/sys_watchdog.sh

#!/bin/bash
# ==============================================================================
# HOMELAB SYSTEM WATCHDOG & AUTO-TUNER
# ==============================================================================

TELEGRAM_BOT_TOKEN="YOUR_TELEGRAM_BOT_TOKEN"
TELEGRAM_CHAT_ID="YOUR_TELEGRAM_CHAT_ID"

send_alert() {
    local message="$1"
    if [ -n "$TELEGRAM_BOT_TOKEN" ]; then
        curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
            -F chat_id="${TELEGRAM_CHAT_ID}" \
            -F text="⚠️ <b>SYSTEM WATCHDOG TRIGGERED</b>
🖥 Host: YOUR_SERVER_NAME
$message" \
            -F parse_mode="HTML" > /dev/null
    fi
}

# 1. AUTO-TUNE DISK CACHE (Drop page cache if I/O utilization is >= 90%)
DISK_UTIL=$(iostat -xz 1 2 | awk '/^sda|^nvme0n1/ {print $NF}' | tail -n 1 | cut -d'.' -f1)

if [ -n "$DISK_UTIL" ] && [ "$DISK_UTIL" -ge 90 ]; then
    sync
    echo 3 > /proc/sys/vm/drop_caches
    TOP_IO=$(iotop -b -n 1 -o | head -n 5)
    send_alert "<b>Disk Saturation:</b> ${DISK_UTIL}% utilization. 
<b>Action Taken:</b> Dropped filesystem caches.
<b>Top I/O Consumers:</b>
<pre>$TOP_IO</pre>"
fi

# 2. AUTO-CLEAN SYSTEMD JOURNAL BLOAT IF DISK SPACE RUNS LOW
DISK_FREE=$(df / | awk 'NR==2 {print $4}') # Free space in KB
if [ "$DISK_FREE" -lt 5242880 ]; then # Less than 5GB free
    journalctl --vacuum-size=200M
    docker system prune -f
    send_alert "<b>Low Disk Space Warning:</b> Less than 5GB remaining. Cleaned systemd journals and pruned Docker cache."
fi

Make it executable and add it to your root crontab to run every 15 minutes:

sudo chmod +x /root/sys_watchdog.sh
# Add to crontab: */15 * * * * /root/sys_watchdog.sh

Phase 22: Upgrading to the Prometheus React UI

The modern Prometheus React UI includes syntax highlighting and auto-complete, making it the perfect scratchpad for testing telemetry math before building your final Grafana panels.

# Run the Debian helper script to pull the React assets
sudo /usr/share/prometheus/install-ui.sh
sudo systemctl restart prometheus

Navigate to http://<YOUR_PROMETHEUS_IP>:9090/graph to use the new interface.

Phase 23: Managing the ARP Cache (ip neighbor)

If a device switches IPs on your network but your server cannot connect, you likely have a stale ARP cache. View current mappings:

ip n

If a mapping is corrupted, flush the entire cache to force the Linux kernel to re-discover hardware dynamically:

sudo ip neighbor flush all

Phase 24: Resolving VNC Desktop Crashes (Signal 6 & Polkit)

If you see a Polkit/D-Bus permissions error like no session for pid, purge conflicting desktop packages and isolate XFCE:

sudo apt purge lxde-core lxpanel lxterminal xrdp -y
sudo apt autoremove --purge -y
sudo apt install --reinstall xfce4 xfce4-goodies -y

If Snap applications (like Chromium) crash your VNC session, bypass the cgroup slice by exporting the runtime directory before execution: export XDG_RUNTIME_DIR=/run/user/$(id -u).

Phase 26: Preventing System Bloat & Log Rotations

Limit systemd journal logs to a manageable size to protect your storage drive:

sudo journalctl --vacuum-size=500M

Enforcing Docker Log Limits

By default, Docker container logs grow infinitely. Force Docker to rotate logs by creating a global daemon configuration:

sudo nano /etc/docker/daemon.json

Paste the following JSON to restrict containers to three 10MB log files max:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Restart the Docker engine: sudo systemctl restart docker.

Phase 29: Securing Kali Linux & Nmap Discovery

When deploying penetration testing distributions like Kali Linux within your lab, remote access protocols are disabled by default to minimize the attack surface. Enable them securely and utilize Nmap for network reconnaissance.

1. Bootstrap the OpenSSH Server

sudo apt update && sudo apt install openssh-server -y
sudo systemctl enable --now ssh
sudo systemctl status ssh --no-pager

2. Nmap Host Discovery & Optimization

Decoy Scans (Evasion): Mask your scanning origin: nmap -sS -D 10.1.0.1 <TARGET_IP>
Skip Host Discovery (No Ping): Bypass ICMP firewalls: nmap -Pn <TARGET_IP>
Aggressive Assessments: sudo nmap -A -v -T4 192.168.1.*

Phase 30: Automated Debian & Kali Daily Diagnostics

Consolidate your daily security and performance checks into a unified maintenance shell utility.

#!/bin/bash
# DEBIAN / KALI LINUX DAILY MAINTENANCE UTILITY
if [ "$EUID" -ne 0 ]; then
 echo "[-] Error: Please run this diagnostic script with sudo privileges."
 exit 1
fi

echo "======================================================================"
echo " 🔄 PHASE 1: PACKAGE MANAGER RESYNCHRONIZATION & REPAIR"
echo "======================================================================"
sudo apt update && sudo apt upgrade -y
sudo apt --fix-broken install -y
sudo apt autoremove -y

echo "======================================================================"
echo " 📡 PHASE 2: NETWORK INTERFACES & SECURE TOPOLOGY"
echo "======================================================================"
echo "[+] VNC Socket Bindings:"
sudo netstat -ptnl | grep vnc
echo "[+] Link Layer IP Routing Tables:"
ip addr

echo "======================================================================"
echo " ⚙️ PHASE 3: OS ARCHITECTURE & ENVIRONMENT"
echo "======================================================================"
sudo hostnamectl
sudo lsb_release -a

echo "======================================================================"
echo " 📊 PHASE 4: DISK MANAGEMENT & CONTAINER METRICS"
echo "======================================================================"
if command -v smbstatus &> /dev/null; then
 sudo smbstatus --shares | column -t
fi
lsblk

echo "======================================================================"
echo " 🛠 PHASE 5: HARDWARE FOOTPRINT & PERFORMANCE"
echo "======================================================================"
lscpu
vmstat

Save as dailyscri.sh, apply permissions (chmod +x dailyscri.sh), and execute via sudo ./dailyscri.sh.

Phase 31: Advanced Wazuh Administration & Fixes

Maintaining a Wazuh SIEM deployment occasionally requires manual intervention to break APT package loops.

1. Fixing Broken Wazuh-Manager Uninstallation

If purging the wazuh-manager throws exit status errors, bypass the broken maintainer scripts manually:

sudo nano /var/lib/dpkg/info/wazuh-manager.prerm
# Replace contents with: #!/bin/sh \n exit 0

sudo nano /var/lib/dpkg/info/wazuh-manager.postrm
# Replace contents with: #!/bin/sh \n exit 0

sudo apt-get autoremove -y && sudo apt-get clean
sudo rm -rf /var/ossec
sudo dpkg --purge wazuh-manager

2. Enabling Anonymous Login on Wazuh Dashboard (Lab Only)

Open /etc/wazuh-indexer/opensearch-security/config.yml and set anonymous_auth_enabled: true. Map "opendistro_security_anonymous" to all_access roles, run the securityadmin.sh sync script, and add opensearch_security.auth.anonymous_auth_enabled: true to the dashboard config.

Phase 32: Securing Docker Apps with Nginx Reverse Proxy

Lock down custom web apps by restricting Docker to localhost and wrapping the traffic in an Nginx proxy protected by HTTP Basic Authentication.

#!/bin/bash
# 1. Install Nginx & Apache Utilities
sudo apt update && sudo apt install nginx apache2-utils -y

# 2. Setup Nginx Password File
sudo htpasswd -c /etc/nginx/.htpasswd admin

# 3. Pull/Update Application & Deploy Container (e.g., Bookmark-Doc-app)
# Ensure docker-compose.yml maps ports as: "127.0.0.1:5000:5000"

# 4. Setup Nginx Proxy Configuration
sudo bash -c 'cat > /etc/nginx/sites-available/python_proxy << "EOF"
server {
    listen 9000;
    server_name _;
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
EOF'

# 5. Enable site and restart Nginx
sudo ln -s /etc/nginx/sites-available/python_proxy /etc/nginx/sites-enabled/
sudo ufw allow 9000/tcp
sudo nginx -t
sudo systemctl restart nginx

Phase 33: Self-Hosting Navidrome with an External NAS

Deploy Navidrome using Docker Compose, mount an external NAS folder for music, and safely store the database locally.

services:
  navidrome:
    image: deluan/navidrome:latest
    user: 1000:1000
    ports:
      - "4533:4533"
    restart: always
    environment:
      ND_LOGLEVEL: info 
      ND_SCANSCHEDULE: "@every 1h"
    volumes:
      - "/home/<YOUR_USERNAME>/navidrome_data:/data"
      - "/mnt/external/NAS/SharedMusic:/music:ro"

Phase 34: Securing Navidrome with a Tailscale Funnel & HA Kill Switch

To listen to your music on strict corporate networks, use Tailscale Funnel. We will build a physical "Kill Switch" inside Home Assistant to toggle the gateway.

The Home Assistant YAML Configuration

command_line:
  - switch:
      name: "Navidrome Public Tunnel"
      unique_id: navidrome_public_tunnel
      command_on: "ssh -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /config/.ssh/id_rsa <YOUR_USERNAME>@<YOUR_HOST_IP> 'sudo tailscale funnel --bg 4533'"
      command_off: "ssh -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /config/.ssh/id_rsa <YOUR_USERNAME>@<YOUR_HOST_IP> 'sudo tailscale serve reset'"
      command_state: "ssh -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /config/.ssh/id_rsa <YOUR_USERNAME>@<YOUR_HOST_IP> 'sudo tailscale funnel status | grep 4533'"
      icon: mdi:music-network

Phase 35: Automating the Funnel via Telegram & Safety Timers

Build a two-way Telegram alert system with an interactive inline button, backed by a 2-hour "dead man's switch" inside Home Assistant.

alias: "Security: Auto-Close Navidrome Tunnel (2 Hours)"
mode: single
trigger:
  - platform: state
    entity_id: switch.navidrome_public_tunnel
    to: "on"
    for:
      hours: 2
action:
  - action: switch.turn_off
    target:
      entity_id: switch.navidrome_public_tunnel

Phase 36: Proxmox USB Backups (The "Host Owns It" Architecture)

To avoid target is busy backup errors in Proxmox, mount the physical USB drive on the Proxmox host and share it to the VM via NFS.

# On Proxmox Host:
apt install nfs-kernel-server -y
nano /etc/exports
# Add: /mnt/pve/usb-storage <VM_IP>(rw,sync,no_subtree_check,no_root_squash,fsid=1)
exportfs -arv

# Inside VM (/etc/fstab):
# <PROXMOX_IP>:/mnt/pve/usb-storage /mnt/external nfs defaults,_netdev,nofail 0 0

Phase 37: Troubleshooting Wazuh Startup Timeouts

Increase the systemd timeout threshold to 5 minutes to prevent orphaned processes on resource-constrained VMs:

sudo mkdir -p /etc/systemd/system/wazuh-manager.service.d
echo -e "[Service]\nTimeoutStartSec=300" | sudo tee /etc/systemd/system/wazuh-manager.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart wazuh-manager

Phase 38: Integrating Wazuh IT Hygiene Dashboards into Grafana

Extract endpoint telemetry into Grafana by generating long-lived JWT Bearer Tokens (1 year expiry) for the Wazuh REST API and deploying the Infinity Data Source plugin.

Phase 39 & 46: AI-Powered SOC Triage (LiteLLM Gateway & Real-Time Fast Lane)

Piping raw SIEM alerts directly into an LLM causes prompt context overflows and high API costs. Deploying a centralized LiteLLM Proxy acts as a unified OpenAI-compatible gateway that translates alert streams across local Ollama instances and cloud models.

1. Deploying the LiteLLM Gateway

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    ports:
      - "4000:4000"
    volumes:
      - ./litellm/config.yaml:/app/config.yaml
    environment:
      - LITELLM_MASTER_KEY=YOUR_LITELLM_API_KEY
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    restart: unless-stopped

2. Routing Configuration (`./litellm/config.yaml`)

model_list:
  - model_name: vibe-coder
    litellm_params:
      model: groq/llama-3.3-70b-versatile
      api_key: "YOUR_UPSTREAM_PROVIDER_API_KEY"

  - model_name: qwen-cluster
    litellm_params:
      model: ollama/qwen2.5:1.5b
      api_base: "http://<YOUR_LOCAL_IP>:11434"

general_settings:
  master_key: YOUR_LITELLM_API_KEY

3. Wazuh OSSEC Integration Hook

Append this integration block inside /var/ossec/etc/ossec.conf to execute triage on Level 12+ events:

  <integration>
    <name>custom-llm-telegram</name>
    <level>12</level>
    <alert_format>json</alert_format>
  </integration>

4. Real-Time Triage & Telegram Push Script

Save to /var/ossec/integrations/custom-llm-telegram, make executable (chmod 750), and assign ownership (chown root:wazuh):

#!/usr/bin/env python3
import sys, json, urllib.request

LITELLM_URL = "http://<YOUR_LITELLM_IP>:4000/v1/chat/completions"
LITELLM_KEY = "YOUR_LITELLM_API_KEY"
TELEGRAM_BOT_TOKEN = "YOUR_TELEGRAM_BOT_TOKEN"
TELEGRAM_CHAT_ID = "YOUR_TELEGRAM_CHAT_ID"
FAST_MODEL = "vibe-coder"

def send_telegram(text):
    url = f"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendMessage"
    payload = {"chat_id": TELEGRAM_CHAT_ID, "text": text, "parse_mode": "HTML"}
    req = urllib.request.Request(url, data=json.dumps(payload).encode("utf-8"), headers={"Content-Type": "application/json"})
    urllib.request.urlopen(req, timeout=15)

def analyze_with_llm(alert_json):
    system_prompt = "You are an automated SOC triage analyst. Analyze this Level 12+ security alert. Provide a concise analysis in 3 bullet points: 1. Threat Nature & Impact 2. Likely Attack Vector / Indicator 3. Recommended Containment Step. Be direct."
    prompt = f"Wazuh Alert Data:\n{json.dumps(alert_json, indent=2)}"
    payload = {"model": FAST_MODEL, "messages": [{"role": "system", "content": system_prompt}, {"role": "user", "content": prompt}], "temperature": 0.2}
    req = urllib.request.Request(LITELLM_URL, data=json.dumps(payload).encode("utf-8"), headers={"Content-Type": "application/json", "Authorization": f"Bearer {LITELLM_KEY}"})
    with urllib.request.urlopen(req, timeout=60) as resp:
        result = json.loads(resp.read().decode("utf-8"))
        return result["choices"][0]["message"]["content"]

def main():
    if len(sys.argv) < 2: sys.exit(1)
    with open(sys.argv[1], "r") as f: alert_data = json.load(f)
    
    rule = alert_data.get("rule", {})
    agent = alert_data.get("agent", {})
    level = rule.get("level", 0)
    if level < 12: sys.exit(0)

    try:
        triage_report = analyze_with_llm(alert_data)
    except Exception as e:
        triage_report = f"⚠️ LLM Analysis Failed: {str(e)}"

    safe_report = triage_report.replace('<', '&lt;').replace('>', '&gt;')
    msg = f"🚨 <b>CRITICAL WAZUH ALERT (Level {level})</b>\n\n<b>Host:</b> <code>{agent.get('name', 'Unknown')}</code>\n<b>Rule:</b> <code>{rule.get('id', 'N/A')}</code> - {rule.get('description', 'N/A')}\n\n<b>AI Triage Analysis:</b>\n{safe_report}"
    send_telegram(msg)

if __name__ == "__main__": main()

Phase 41: Advanced Media Management (Immich & Jellyfin)

Customizing the Jellyfin UI

Inject custom CSS to personalize the login screens using modern accent colors:

:root {
  --accent1-light: #00a4dc; /* Vibrant Blue */
  --accent2-light: #aa5cc3; /* Deep Purple */
}
.paper-icon-button-light:hover, .raised, .button-submit {
  background: linear-gradient(135deg, var(--accent1-light), var(--accent2-light)) !important;
  color: #ffffff !important;
}

Phase 42: Lightweight Storage & Custom Web Apps

A Raspberry Pi running OpenMediaVault (OMV) acts as a highly energy-efficient NAS. It is also the perfect host for custom Python web apps (like Streamlit dashboards) placed behind an Nginx reverse proxy.

# OMV Docker Pruning Maintenance
docker image prune -a -f
docker volume prune -f

Phase 43: Centralized Telegram Alerts for Homelab Cronjobs

Instead of setting up a heavy centralized monitoring server to track automated background tasks across multiple VMs and Raspberry Pis, use this lightweight Bash Wrapper Agent. Placed on each node, it wraps your existing crontab jobs and pushes formatted execution metrics, exit codes, and error logs directly to your phone.

sudo nano /usr/local/bin/cron_agent.sh

Paste the following code, replacing the credentials with your Telegram Bot Token and Chat ID:

#!/bin/bash
# ==========================================
# Telegram Cronjob Monitoring Agent
# ==========================================

BOT_TOKEN="YOUR_TELEGRAM_BOT_TOKEN"
CHAT_ID="YOUR_TELEGRAM_CHAT_ID"

# Parse arguments
ALERT_SUCCESS=true
if [ "$1" == "--fail-only" ]; then
    ALERT_SUCCESS=false
    shift
fi

SERVER_NAME="$1"
JOB_NAME="$2"
shift 2
COMMAND="$@"

# Start execution and timer
START_TIME=$(date +%s)
OUTPUT=$($COMMAND 2>&1)
EXIT_CODE=$?
END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))

# Determine Status
if [ $EXIT_CODE -eq 0 ]; then
    if [ "$ALERT_SUCCESS" = false ]; then
        exit 0
    fi
    STATUS="✅ SUCCESS"
else
    STATUS="❌ FAILED (Code: $EXIT_CODE)"
fi

# Format the Telegram Message
MESSAGE=$(cat <<EOF
$STATUS
🖥 Host: $SERVER_NAME
🛠 Job: $JOB_NAME
⏱ Duration: ${DURATION}s

📄 Output:
$(echo "$OUTPUT" | tail -n 15)
EOF
)

# Send to Telegram
curl -s -X POST "https://api.telegram.org/bot${BOT_TOKEN}/sendMessage" \
    -F chat_id="${CHAT_ID}" \
    -F text="$MESSAGE" > /dev/null

Secure the file: sudo chmod +x /usr/local/bin/cron_agent.sh. You can now wrap any cron job using this syntax: /usr/local/bin/cron_agent.sh [--fail-only] "Host Name" "Job Name" /path/to/script.sh.

Phase 44: Deploying RustDesk over Tailscale

Self-hosting your remote desktop infrastructure using RustDesk provides complete control over your privacy. Running it strictly over your Tailscale mesh network creates an encrypted, private administration backbone.

1. The Docker Compose Configuration

services:
  hbbs:
    container_name: hbbs
    image: rustdesk/rustdesk-server:latest
    environment:
      - ALWAYS_USE_RELAY=Y
    command: hbbs -r <YOUR_VPN_IP>:21117 -k _
    volumes:
      - ./data:/root
    network_mode: "host"
    depends_on:
      - hbbr
    restart: unless-stopped

  hbbr:
    container_name: hbbr
    image: rustdesk/rustdesk-server:latest
    command: hbbr -k _
    volumes:
      - ./data:/root
    network_mode: "host"
    restart: unless-stopped

2. Extracting the Encryption Key

Because the RustDesk image is "distroless", you must copy the auto-generated ED25519 key out to the host to read it:

sudo docker compose up -d
sudo docker cp hbbs:/root/id_ed25519.pub /tmp/id_ed25519.pub
cat /tmp/id_ed25519.pub

Copy this key into your RustDesk client settings under the Key field.

Phase 45: Glances Systemd Service & Hardware Sensors

Configure Glances to run permanently as a lightweight background web server to monitor CPU and motherboard thermals.

sudo apt update && sudo apt install lm-sensors glances -y
sudo sensors-detect --auto
sudo nano /etc/systemd/system/glances.service

[Unit]
Description=Glances Web Server Daemon
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/glances -w -B 0.0.0.0 -p 61208
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Initialize the service: sudo systemctl enable --now glances.service.

Phase 47: Wazuh Batch Processing & ISO 27001 Compliance Auditing

To avoid hitting LLM context limits, the "Batch Processing Lane" uses a Python script to query OpenSearch, minify the JSON data down to aggregated counters, and process the results locally using a privacy-first AI swarm.

1. The Daily IT Hygiene Digest

sudo nano /usr/local/bin/wazuh-daily-digest.py

#!/usr/bin/env python3
import urllib.request, json, ssl, base64, sys
from collections import Counter

INDEXER_URL = "https://127.0.0.1:9200/wazuh-alerts-*/_search"
INDEXER_USER = "admin"
INDEXER_PASS = "YOUR_INDEXER_PASSWORD"
LITELLM_URL = "http://<YOUR_LITELLM_IP>:4000/v1/chat/completions"
LITELLM_KEY = "YOUR_LITELLM_API_KEY"
TELEGRAM_BOT_TOKEN = "YOUR_TELEGRAM_BOT_TOKEN"
TELEGRAM_CHAT_ID = "YOUR_TELEGRAM_CHAT_ID"
LOCAL_MODEL = "qwen-cluster"

def fetch_wazuh_alerts():
    query = {
        "query": {"bool": {"must": [{"range": {"timestamp": {"gte": "now-24h", "lte": "now"}}}, {"range": {"rule.level": {"gte": 5}}}]}},
        "size": 1000, "sort": [{"rule.level": "desc"}]
    }
    ctx = ssl.create_default_context()
    ctx.check_hostname = False; ctx.verify_mode = ssl.CERT_NONE
    req = urllib.request.Request(INDEXER_URL, data=json.dumps(query).encode("utf-8"), headers={"Content-Type": "application/json"})
    auth_b64 = base64.b64encode(f"{INDEXER_USER}:{INDEXER_PASS}".encode()).decode()
    req.add_header("Authorization", f"Basic {auth_b64}")
    with urllib.request.urlopen(req, context=ctx) as resp:
        return json.loads(resp.read().decode("utf-8"))

def minify_data(data):
    hits = data.get("hits", {}).get("hits", [])
    if not hits: return None
    rule_counts = Counter()
    for hit in hits:
        source = hit.get("_source", {})
        rule_counts[f"Level {source.get('rule', {}).get('level', 0)}: {source.get('rule', {}).get('description', 'Unknown')}"] += 1
        
    summary = "=== 24H ALERT DATA ===\nTop Alert Types:\n"
    for rule, count in rule_counts.most_common(10): summary += f"- {rule} (Triggered {count} times)\n"
    return summary

def analyze_with_llm(minified_data):
    system_prompt = "You are a SOC manager writing a daily IT Hygiene digest. Review the provided alert counts for the last 24 hours. Provide a summary explaining overall network health and specific remediation actions. Output plain text only."
    payload = {"model": LOCAL_MODEL, "messages": [{"role": "system", "content": system_prompt}, {"role": "user", "content": minified_data}], "temperature": 0.3}
    req = urllib.request.Request(LITELLM_URL, data=json.dumps(payload).encode("utf-8"), headers={"Content-Type": "application/json", "Authorization": f"Bearer {LITELLM_KEY}"})
    with urllib.request.urlopen(req, timeout=300) as resp:
        return json.loads(resp.read().decode("utf-8"))["choices"][0]["message"]["content"]

def send_telegram(analysis):
    url = f"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendMessage"
    clean_analysis = analysis.replace('<', '&lt;').replace('>', '&gt;').replace('*', '').replace('#', '')
    msg = f"📊 <b>Daily IT Hygiene Digest</b>\n\n<b>Swarm Analysis:</b>\n{clean_analysis}"
    payload = {"chat_id": TELEGRAM_CHAT_ID, "text": msg, "parse_mode": "HTML"}
    req = urllib.request.Request(url, data=json.dumps(payload).encode("utf-8"), headers={"Content-Type": "application/json"})
    urllib.request.urlopen(req, timeout=15)

def main():
    raw_data = fetch_wazuh_alerts()
    minified_data = minify_data(raw_data)
    if minified_data: send_telegram(analyze_with_llm(minified_data))

if __name__ == "__main__": main()

2. Understanding ISO 27001 Compliance Checking

Wazuh natively maps its internal security rules to major regulatory frameworks, including ISO/IEC 27001. When an endpoint generates an alert, Wazuh automatically attaches relevant control tags directly inside the JSON alert object under rule.compliance.iso_27001.

To isolate and audit these compliance gaps over a weekly cycle, we adjust our OpenSearch query to look specifically for logs containing active ISO control triggers, and parse out the exact host agents responsible:

# 1. OpenSearch Query Adjustment (Look back 7 days for ISO compliance flags)
{"query": {"bool": {"must": [{"range": {"timestamp": {"gte": "now-7d", "lte": "now"}}}, {"wildcard": {"rule.compliance.iso_27001": "*"}}]}}, "size": 1000}

# 2. Minification Loop Adjustment (Extract specific control tags and host names)
for hit in hits:
    source = hit.get("_source", {})
    agent = source.get("agent", {}).get("name", "Unknown Host")
    for tag in source.get("rule", {}).get("compliance", {}).get("iso_27001", []):
        cve_list[f"ISO 27001 Control {tag} triggered on host '{agent}'"] += 1

Phase 48: Immich Photo Server & Hardware Acceleration

Deploying Immich natively requires correctly passing your iGPU through to the containers to enable VA-API video transcoding and OpenVINO machine learning tasks. It also requires the pgvectors extension inside PostgreSQL.

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    devices:
      - /dev/dri:/dev/dri # Pass iGPU for VA-API video transcoding
    volumes:
      - ${UPLOAD_LOCATION}:/data
      - /mnt/nas_photos:/usr/src/app/external/nas_photos:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-openvino
    devices:
      - /dev/dri:/dev/dri # Pass iGPU for OpenVINO acceleration
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: valkey/valkey:8-alpine
    restart: always

  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    shm_size: 128mb
    restart: always

volumes:
  model-cache:

Phase 50: Monitoring Docker Containers with Wazuh

Wazuh uses a built-in docker-listener wodle to monitor container lifecycle events (start, stop, pause, pull).

sudo apt update && sudo apt install python3-pip python3-docker -y
sudo usermod -aG docker wazuh

Open /var/ossec/etc/ossec.conf and add:

<wodle name="docker-listener">
  <disabled>no</disabled>
  <interval>10m</interval>
  <attempts>5</attempts>
  <run_on_start>yes</run_on_start>
</wodle>

Restart the daemon: sudo systemctl restart wazuh-manager.

Phase 51: Essential Sysadmin CLI Extras & MacBook Drivers

Visual System Fetch: Use fastfetch --logo none.
Terminal Web Browser: sudo apt install lynx -y
Flatpak Integration: sudo apt install flatpak gnome-software-plugin-flatpak

MacBook Pro Linux Fixes (HFS+ & Drivers)

# Grant Write Access to macOS HFS+ Drives
sudo apt install hfsprogs
sudo mount -t hfsplus -o rw,force /dev/sdX /mnt/mac_drive/

# Install Broadcom Wi-Fi Drivers
sudo apt install firmware-b43-installer broadcom-sta-dkms -y

# Fix Fan Curves & Power Management
sudo apt install -y tlp tlp-rdw mbpfan
sudo systemctl enable --now tlp mbpfan

# Fix Apple Backlit Keyboard
sudo apt install pommed

Phase 52: Native Remote Desktop Access (Remmina)

If you deployed the full Ubuntu GNOME desktop and want to connect to it locally using the built-in screen sharing protocol, open Settings > Sharing, enable Remote Desktop and Remote Control, and explicitly set a VNC Auth password. Then connect from your client using Remmina.

Phase 53: Infrastructure as Code - Automating Nodes with Ansible

Instead of running manual configuration commands on each node individually, Ansible allows you to define your desired server state in centralized YAML playbooks and push changes across your entire fleet instantly.

1. Setting Up the Control Node

sudo apt update && sudo apt install ansible -y

2. Defining the Inventory File (`hosts.ini`)

[proxmox_hosts]
pve-node ansible_host=<PROXMOX_IP> ansible_user=root

[docker_nodes]
server1454 ansible_host=<SERVER_IP> ansible_user=server1454

[pis]
node_pi ansible_host=<RASPI_IP> ansible_user=pi

[all:vars]
ansible_ssh_private_key_file=~/.ssh/id_ed25519

3. Writing a Master Setup Playbook (`site.yml`)

---
- name: Configure and Harden Homelab Nodes
  hosts: all
  become: yes
  tasks:
    - name: Update apt repo cache
      apt:
        update_cache: yes
        cache_valid_time: 3600

    - name: Install essential sysadmin utilities
      apt:
        name:
          - btop
          - curl
          - git
          - ufw
          - cifs-utils
        state: present

    - name: Ensure UFW firewall is active
      ufw:
        state: enabled
        policy: allow

4. Executing the Playbook

ansible all -i hosts.ini -m ping
ansible-playbook -i hosts.ini site.yml

Phase 54: The Ultimate Homelab Auto-Backup Pipeline

A true disaster recovery plan requires a "zero-touch" automated backup system that captures everything, prunes old files to save storage, and alerts you upon completion.

1. Create the Master Backup Script

sudo nano /root/homelab_backup.sh

#!/bin/bash
if [ "$EUID" -ne 0 ]; then echo "[-] Please run as root"; exit 1; fi

BACKUP_BASE="/mnt/external/backup"
BACKUP_DIR="$BACKUP_BASE/homelab_$(date +%F)"
mkdir -p "$BACKUP_DIR"

echo "Backing up Docker Data..."
tar -czf "$BACKUP_DIR/docker_app_data_backup.tar.gz" /home/<YOUR_USERNAME>/homepage/config /home/<YOUR_USERNAME>/jellyfin/config /home/<YOUR_USERNAME>/navidrome_data /docker/netalertx/data 2>/dev/null
tar -czf "$BACKUP_DIR/portainer_stacks_and_db.tar.gz" /var/lib/docker/volumes/portainer_data/_data/ 2>/dev/null
tar -czf "$BACKUP_DIR/standalone_docker_apps.tar.gz" /home/<YOUR_USERNAME>/docker/Bookmark-Doc-app/ 2>/dev/null

echo "Backing up Wazuh & AI SOC..."
tar -czf "$BACKUP_DIR/wazuh_config_backup.tar.gz" /var/ossec/etc/ossec.conf /var/ossec/etc/rules/local_rules.xml /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml 2>/dev/null
tar -czf "$BACKUP_DIR/ai_triage_scripts_backup.tar.gz" /var/ossec/integrations/custom-llm-telegram /usr/local/bin/wazuh-daily-digest.py 2>/dev/null

echo "Backing up Telemetry & Reverse Proxies..."
tar -czf "$BACKUP_DIR/telemetry_backup.tar.gz" /etc/prometheus/prometheus.yml /etc/prometheus/alert.rules.yml /etc/alertmanager/alertmanager.yml 2>/dev/null
tar -czf "$BACKUP_DIR/nginx_proxy_backup.tar.gz" /etc/nginx/sites-available/ /etc/nginx/.htpasswd 2>/dev/null

echo "Backing up Scripts & Systemd Services..."
tar -czf "$BACKUP_DIR/custom_scripts_vnc_backup.tar.gz" /home/<YOUR_USERNAME>/.config/tigervnc/xstartup /home/<YOUR_USERNAME>/.bashrc /home/<YOUR_USERNAME>/setup_screen.sh /root/sys_watchdog.sh /usr/local/bin/cron_agent.sh 2>/dev/null
tar -czf "$BACKUP_DIR/systemd_services_backup.tar.gz" /etc/systemd/system/vncserver@.service /etc/systemd/system/scaphandre.service /etc/systemd/system/alertmanager.service /etc/systemd/system/glances.service 2>/dev/null

echo "Cleaning up backups older than 30 days..."
find "$BACKUP_BASE" -type d -name "homelab_*" -mtime +30 -exec rm -rf {} +
echo "Backup Complete!"

2. Schedule with Telegram Alerts

sudo chmod +x /root/homelab_backup.sh
sudo crontab -e
# Add: 0 3 * * 0 /usr/local/bin/cron_agent.sh "<YOUR_SERVER_NAME>" "Weekly Backup" /root/homelab_backup.sh

Where to Go Next: Future Home Lab Expansions

Once you have implemented the entire master blueprint, your infrastructure will be running like a well-oiled, enterprise-grade machine. But the learning never stops.

1. Reverse Proxies & SSL Certificates

Deploying a reverse proxy is the next critical step. Tools like Traefik or Nginx Proxy Manager integrate directly with Docker to route traffic and provision free SSL certificates via Let's Encrypt. (Refer to Phase 32 for the Nginx manual deployment script).

2. Automated Backup Pipelines

Consider deploying automated solutions to encrypt and push your configuration directories to offsite cloud storage (like AWS S3) nightly. (Refer to Phase 54 for your local disaster recovery tarball scripts).

3. Transitioning to Infrastructure as Code (IaC)

The next evolution is managing your entire host configuration using automated orchestration frameworks. (Refer to Phase 53 for your Ansible playbook strategy).

Appendices A-G: Supplemental Reference Material

Appendix A: Complete Automation Scripts

1. Automated DNS Auditor (dns_audit.sh): Use this script to benchmark your local system resolver against global Anycast networks.

#!/bin/bash
if ! command -v dig &> /dev/null; then sudo apt update -qq && sudo apt install dnsutils -y -qq; fi
echo "System Default Link: $(dig example.com | grep "Query time" | awk '{print $4, $5}')"
echo "Cloudflare (1.1.1.1): $(dig example.com @1.1.1.1 | grep "Query time" | awk '{print $4, $5}')"
echo "Google (8.8.8.8): $(dig example.com @8.8.8.8 | grep "Query time" | awk '{print $4, $5}')"

2. All-In-One Display Controller (setup_screen.sh): Detects sysfs backlight interfaces and injects screen aliases.

#!/bin/bash
BACKLIGHT_DIR="/sys/class/backlight"
INTERFACE=$(ls "$BACKLIGHT_DIR" | head -n 1)
BACKLIGHT_PATH="$BACKLIGHT_DIR/$INTERFACE"
MAX_BRIGHTNESS=$(cat "$BACKLIGHT_PATH/max_brightness")

REAL_USER=${SUDO_USER:-$USER}
BASHRC_PATH="$(eval echo ~$REAL_USER)/.bashrc"
echo "alias screenoff='echo 0 | sudo tee $BACKLIGHT_PATH/brightness'" >> "$BASHRC_PATH"
echo "alias screenon='echo $MAX_BRIGHTNESS | sudo tee $BACKLIGHT_PATH/brightness'" >> "$BASHRC_PATH"

Appendix B: Bridging Isolated Subnets via SSH Tunnels

When running isolated VMs on Proxmox, bridge Prometheus to host data via persistent SSH tunnels.

[Unit]
Description=SSH Tunnel to Proxmox Scaphandre
After=network.target ssh.service

[Service]
User=<YOUR_VM_USER>
ExecStart=/usr/bin/ssh -N -T -o ServerAliveInterval=60 -o ExitOnForwardFailure=yes -L 8081:localhost:8081 root@<PROXMOX_HOST_IP>
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Appendix C: Advanced Networking & VNC Polkit Overrides

Fixing the "Not Authorized" Polkit Error over SSH/VNC sessions for nmcli commands:

sudo nano /etc/polkit-1/rules.d/99-networkmanager-vnc.rules

polkit.addRule(function(action, subject) {
    if (action.id.indexOf("org.freedesktop.NetworkManager.") == 0 &&
        subject.isInGroup("sudo")) {
        return polkit.Result.YES;
    }
});

Appendix D: The Unified Docker Compose Stack

Consolidate your core services into a single docker-compose.yml deployment, utilizing a root .env file for PUID, PGID, and TZ values, and pointing global volumes to secure CIFS/SMB mounts.

1. The Master .env File
Create this file in your root deployment directory (e.g., ~/homelab/.env). It centralizes all your variables so you never have to hardcode passwords into the YAML structure.

# ==============================================================================
# HOMELAB MASTER ENVIRONMENT VARIABLES (.env)
# ==============================================================================

# SYSTEM IDENTITY
PUID=1000
PGID=1000
TZ=Europe/London
HOST_IP=<YOUR_HOST_IP>
VPN_IP=<YOUR_TAILSCALE_IP>

# NAS & STORAGE CREDENTIALS
NAS_USER=<YOUR_NAS_USERNAME>
NAS_PASS=<YOUR_NAS_PASSWORD>
NAS_IP=<YOUR_NAS_IP>
NAS_SHARE_NAME=SharedMedia

# PI-HOLE SECURITY
PIHOLE_WEBPASSWORD=<YOUR_SECURE_PASSWORD>

# SPEEDTEST TRACKER DATABASE
SPEEDTEST_DB_PASSWORD=<YOUR_SPEEDTEST_DB_PASSWORD>
SPEEDTEST_DB_ROOT_PASSWORD=<YOUR_SPEEDTEST_ROOT_PASSWORD>

# IMMICH DATABASE & STORAGE
IMMICH_VERSION=release
UPLOAD_LOCATION=/home/<YOUR_USERNAME>/immich/upload
DB_DATA_LOCATION=/home/<YOUR_USERNAME>/immich/postgres
DB_USERNAME=postgres
DB_PASSWORD=<YOUR_STRONG_ALPHANUMERIC_PASSWORD>
DB_DATABASE_NAME=immich

2. The Unified docker-compose.yml
Place this in the same directory as your .env file. Deploy the entire fleet by running docker compose up -d.

# ==============================================================================
# HOMELAB MASTER ORCHESTRATION STACK
# ==============================================================================
name: homelab-master

services:

  # --- SECTION 1: DASHBOARD, AD-BLOCKING & LOCAL APPS ---
  homepage:
    image: ghcr.io/gethomepage/homepage:latest
    container_name: homepage
    ports:
      - "127.0.0.1:3001:3000"
    volumes:
      - ./homepage/config:/app/config
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
    restart: unless-stopped

  pihole:
    image: pihole/pihole:latest
    container_name: pihole
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "80:80/tcp"
      - "8443:443/tcp"
    environment:
      TZ: ${TZ}
      WEBPASSWORD: ${PIHOLE_WEBPASSWORD}
      FTLCONF_dns_listeningMode: 'ALL'
    volumes:
      - './pihole/etc-pihole:/etc/pihole'
      - './pihole/etc-dnsmasq.d:/etc/dnsmasq.d'
    cap_add:
      - SYS_NICE
    restart: unless-stopped



  # --- SECTION 2: SMART HOME & NETWORK SECURITY ---
  homeassistant:
    image: ghcr.io/home-assistant/home-assistant:stable
    container_name: homeassistant
    network_mode: host
    volumes:
      - ./homeassistant/config:/config
      - /etc/localtime:/etc/localtime:ro
    environment:
      - TZ=${TZ}
    restart: unless-stopped

  netalertx:
    image: ghcr.io/jokob-sk/netalertx:latest
    container_name: NetAlertX
    network_mode: host # Required for local subnet ARP scanning
    volumes:
      - ./netalertx/config:/app/config
      - ./netalertx/data:/app/data
    environment:
      - TZ=${TZ}
      - PORT=20211
    restart: unless-stopped

  # --- SECTION 3: MEDIA & ENTERTAINMENT ---
  plex:
    image: lscr.io/linuxserver/plex:latest
    container_name: plex
    network_mode: host
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=${TZ}
      - VERSION=docker
    volumes:
      - ./plex/config:/config
      - nas_media:/data/nas_shared
    restart: unless-stopped

  navidrome:
    image: deluan/navidrome:latest
    container_name: navidrome
    user: ${PUID}:${PGID}
    ports:
      - "4533:4533"
    environment:
      ND_LOGLEVEL: info 
      ND_SCANSCHEDULE: "@every 1h"
      ND_ENABLEDOWNLOADS: "true"
    volumes:
      - ./navidrome/data:/data
      - nas_media:/music:ro
    restart: unless-stopped

  # --- SECTION 4: SPEEDTEST TRACKER ---
  speedtest-tracker:
    image: lscr.io/linuxserver/speedtest-tracker:latest
    container_name: speedtest-tracker
    ports:
      - "8091:80/tcp"
      - "8444:443/tcp"
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=${TZ}
      - DB_CONNECTION=mysql
      - DB_HOST=speedtest-tracker-db
      - DB_PORT=3306
      - DB_DATABASE=speedtest_tracker
      - DB_USERNAME=speedtest
      - DB_PASSWORD=${SPEEDTEST_DB_PASSWORD}
    volumes:
      - ./speedtest/config:/config
    depends_on:
      - speedtest-tracker-db
    restart: unless-stopped

  speedtest-tracker-db:
    image: mariadb:11
    container_name: speedtest-tracker-db
    environment:
      - MARIADB_DATABASE=speedtest_tracker
      - MARIADB_USER=speedtest
      - MARIADB_PASSWORD=${SPEEDTEST_DB_PASSWORD}
      - MARIADB_ROOT_PASSWORD=${SPEEDTEST_DB_ROOT_PASSWORD}
    volumes:
      - ./speedtest/db:/var/lib/mysql
    restart: unless-stopped

  # --- SECTION 5: IMMICH (HARDWARE ACCELERATED) ---
  immich-server:
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION}
    container_name: immich_server
    devices:
      - /dev/dri:/dev/dri
    ports:
      - '2283:2283'
    volumes:
      - ${UPLOAD_LOCATION}:/data
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - immich-redis
      - immich-postgres
    restart: always

  immich-machine-learning:
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION}-openvino
    container_name: immich_machine_learning
    devices:
      - /dev/dri:/dev/dri
    volumes:
      - immich-model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-redis:
    image: valkey/valkey:8-alpine
    container_name: immich_redis
    restart: always

  immich-postgres:
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0
    container_name: immich_postgres
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    shm_size: 128mb
    restart: always

  # --- SECTION 6: TELEMETRY & REMOTE ACCESS ---
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
      - "8082:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    privileged: true
    cgroupns_mode: host
    restart: always

  hbbs:
    image: rustdesk/rustdesk-server:latest
    container_name: hbbs
    network_mode: "host"
    environment:
      - ALWAYS_USE_RELAY=Y
    command: hbbs -r ${VPN_IP}:21117 -k _
    volumes:
      - ./rustdesk/data:/root
    depends_on:
      - hbbr
    restart: unless-stopped

  hbbr:
    image: rustdesk/rustdesk-server:latest
    container_name: hbbr
    network_mode: "host"
    command: hbbr -k _
    volumes:
      - ./rustdesk/data:/root
    restart: unless-stopped

# --- GLOBAL VOLUMES ---
volumes:
  immich-model-cache:

  nas_media:
    driver: local
    driver_opts:
      type: cifs
      o: "username=${NAS_USER},password=${NAS_PASS},iocharset=utf8,vers=3.0"
      device: "//${NAS_IP}/${NAS_SHARE_NAME}"

Appendix E: Server Maintenance Cheat Sheet

# Update OS and Clean Bloat
sudo apt update && sudo apt upgrade -y && sudo apt autoremove --purge -y
# Prune Docker Caches
docker system prune -a
# Restart VNC Headless Display
sudo systemctl restart vncserver@1.service

Appendix F: Resolving Grafana JSON Validation Errors

If Grafana throws conflicting values "GridLayoutItem" and "GridLayoutImageItem" upon JSON import, open the JSON in a text editor and replace all instances of GridLayoutImageItem with GridLayoutItem. If variables fail, strip the type: "dashboard" parameter.

Appendix G: Security Sanitization Checklist

Ensure no Tailscale 100.x.x.x IPs are visible.
Never publish Telegram bot_token or chat_id strings.
Obscure MAC addresses (e.g., 70:5a:xx:xx:xx) from ip neighbor logs.
Sanitize .env configurations before pushing code to GitHub.

Documentation Framework Finalized

This master guide was compiled directly from real-world terminal logs, debugging sessions, and architectural deployments. From navigating obscure Realtek driver bugs to orchestrating a fully integrated Prometheus telemetry stack, every step here has been battle-tested on live hardware.

Stay Curious. Keep Building. Happy Self-Hosting!

Bonus: Android ADB Debloat Automation Scripts

Safely disable telemetric bloatware, aggressive performance throttlers (like GOS), and sponsored feeds on Samsung One UI and Vivo Funtouch OS via ADB.

1. Vivo Z1x (Funtouch OS 11) Debloat Script

.\adb shell "
cmd appops set com.bbk.cloud RUN_IN_BACKGROUND ignore; pm clear com.bbk.cloud;
cmd appops set com.vivo.pushservice RUN_IN_BACKGROUND ignore; pm clear com.vivo.pushservice;
cmd appops set com.vivo.gamecube RUN_IN_BACKGROUND ignore; pm clear com.vivo.gamecube;
cmd appops set com.vivo.game RUN_IN_BACKGROUND ignore; pm clear com.vivo.game;

cmd package disable-user --user 0 com.bbk.cloud;
cmd package disable-user --user 0 com.vivo.pushservice;
cmd package disable-user --user 0 com.vivo.gamecube;
cmd package disable-user --user 0 com.vivo.game;
"

2. Samsung One UI (Game Optimizing Service & Bloat)

# Remove standard bloat
.\adb shell "
pm uninstall -k --user 0 com.facebook.system;
pm uninstall -k --user 0 com.facebook.appmanager;
pm uninstall -k --user 0 com.facebook.services;
pm uninstall -k --user 0 com.samsung.android.bixby.wakeup;
pm uninstall --user 0 com.samsung.discover;
pm uninstall --user 0 com.sec.android.mimage.avatarstickers;
"

# Freeze & Neutralize Game Optimizing Service (GOS)
.\adb shell pm disable-user --user 0 com.samsung.android.game.gos
.\adb shell cmd appops set com.samsung.android.game.gos RUN_IN_BACKGROUND ignore
.\adb shell pm clear com.samsung.android.game.gos

# Freeze & Neutralize Smart Suggestions
.\adb shell pm disable-user --user 0 com.samsung.android.smartsuggestions
.\adb shell cmd appops set com.samsung.android.smartsuggestions RUN_IN_BACKGROUND ignore
.\adb shell pm clear com.samsung.android.smartsuggestions