{"uuid": "aacd70bf-ca1a-4d39-b2d0-e751c9326fcd", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2021-43267", "type": "seen", "source": "https://gist.github.com/rebizzz/7c343a9d9e9e549943b79f9fa6f6784f", "content": "# Non-GUI Systems Internals Deep Dive: User NixOS vs. CachyOS (Server/Base) vs. Fedora (Server/Base)\n\n## Executive Master Comparison Matrix\n\n| Subsystem / Dimension | User System (NixOS 26.11 Custom) | CachyOS (Server / Base) | Fedora (Server / Base) |\n| :--- | :--- | :--- | :--- |\n| **System Philosophy & State Model** | Declarative Impermanence (`tmpfs` / Btrfs rollback + explicit bind-mounts) | Stateful Rolling Release (Arch baseline, mutable) | Stateful Enterprise Lifecycle (LVM + XFS, point releases) |\n| **Kernel Build & Toolchain** | Linux 7.2.0-NixOS compiled with GCC (`-O2`), standard nixpkgs stdenv | Linux CachyOS compiled with LLVM/Clang (`-O3`), ThinLTO/Full LTO, AutoFDO/BOLT | Linux Upstream compiled with GCC (`-O2`), hardened flags (`-fstack-protector-strong`, `-D_FORTIFY_SOURCE=3`) |\n| **Microarchitecture Target** | Generic `x86-64-v1` / `x86-64-v2` baseline with runtime dispatch | Native optimized repos: `x86-64-v3` (AVX2/BMI2) & `x86-64-v4` (AVX-512/Zen4+) | `x86-64-v2` standard baseline across all official repositories |\n| **Kernel Preemption & HZ** | `CONFIG_PREEMPT_DYNAMIC=y`, `rcutree.enable_rcu_lazy=1` | `CONFIG_PREEMPT_DYNAMIC=y` (Full default), `HZ=500` / `HZ=1000`, low-jitter timer wheels | `CONFIG_PREEMPT_DYNAMIC=y` (Voluntary on Server), `HZ=250` (Server) / `HZ=1000` (Workstation) |\n| **CPU Scheduling** | Native In-Kernel EEVDF (`sched_fair`) | BORE (Burst-Oriented Response Enhancer), `sched_ext` (eBPF: `scx_rusty`, `scx_bpfland`), or tuned EEVDF | Upstream EEVDF (`CONFIG_SCHED_EEVDF`), standard kernel CFS replacement |\n| **Process Nice & Latency Daemon** | `ananicy-cpp` + `ananicy-rules-cachyos` (PID renicing, cgroup standalone) | `ananicy-cpp` + `ananicy-rules-cachyos` (native systemd/cgroup v2 controller integration) | None (Relies strictly on static systemd slices & upstream CFS/EEVDF fair share) |\n| **Frequency & Energy Management** | `auto-cpufreq` (active userspace governor/EPP toggling) + `thermald` | `power-profiles-daemon` (ACPI `platform_profile` & kernel-native `amd_pstate_epp` / `intel_pstate`) | `tuned` (`tuned-adm` profiles on Server) or `power-profiles-daemon` |\n| **Memory Compression (ZRAM)** | Native `zramSwap` (`zstd`, 100% RAM, max 8GB, prio 100), `zswap` forced off | `systemd-zram-generator` (`zstd` / multi-comp `zstd:1`+`zstd:6`, 100\u2013150% RAM, prio 32767) | `zram-generator` (`lzo-rle` or `zstd`, 50\u2013100% RAM, prio 100) |\n| **Memory VM Sysctls** | `swappiness=150` (180 Server), `page-cluster=0`, `vfs_cache_pressure=50` | `swappiness=150` (or 180), `page-cluster=0`, `vfs_cache_pressure=50` | `swappiness=60` (Server) / `100` (Workstation), `page-cluster=0`, `vfs_cache_pressure=100` |\n| **Page Allocator Watermarks** | `watermark_boost_factor=0`, `watermark_scale_factor=125` | `watermark_boost_factor=0`, `watermark_scale_factor=125` (or 200) | `watermark_boost_factor=15000` (kernel default), `watermark_scale_factor=10` (default) |\n| **Dirty Memory Throttling** | Absolute: `dirty_bytes=268MB`, `dirty_background_bytes=67MB`, writeback 15s | Low absolute bytes or reduced ratios (`dirty_ratio=10%`, `dirty_background_ratio=5%`) | Relative: `dirty_ratio=20%`, `dirty_background_ratio=10%`, writeback 5s |\n| **Transparent Hugepages (THP)** | `defrag=defer+madvise`, `khugepaged/max_ptes_none=409`, mode `madvise` | `defrag=defer+madvise`, optimized `khugepaged` scan intervals, mode `always`/`madvise` | `defrag=madvise`, stock `khugepaged` defaults (`max_ptes_none=511`), mode `madvise`/`always` |\n| **Primary Storage Engine** | Btrfs (`zstd:1`, `discard=async`) + ZFS Mirrored Pool (`ashift=12`, `recordsize=1M`) | Btrfs (`zstd:1-3`, `discard=async`) / XFS / ZFS (Prebuilt Cachy repos) | XFS (LVM thin/thick partitions) or standard Btrfs subvolumes |\n| **Block I/O Dispatching** | Strict Udev Tiering (NVMe: `none`, SATA SSD: `mq-deadline`, HDD: `bfq`) | Kernel/Udev Tuned (NVMe: `none`/`kyber`, SATA SSD: `kyber`/`mq-deadline`, HDD: `bfq`) | Upstream Conservative (NVMe: `none`, SATA SSD: `mq-deadline`, HDD: `bfq`/`mq-deadline`) |\n| **TCP Congestion & Qdisc** | TCP BBR + `fq` (Fair Queueing paced socket pacing) | TCP BBRv3 + CAKE / FQ (Custom patched kernel tree) | TCP CUBIC / BBR + `fq_codel` (Standard systemd upstream default) |\n| **Network Socket Optimization** | TFO Mode 3 (Client+Server), MTU Probing (RFC 4821), RFC 1337 TIME_WAIT protection | TFO Mode 3, Low Latency Sysctls, BBRv3 loss rate tuning | Upstream Baseline (Conservative MTU discovery, standard buffer allocations) |\n| **LSM & Access Control** | Modern Modular: Landlock + Yama (Scope 1) + Hardened BPF JIT | DAC Baseline (Optional AppArmor profile, unconfined by default) | SELinux Enforcing (`targeted` Type Enforcement & Multi-Category Security) |\n| **Kernel Exploit Defenses** | Allocator Poisoning (`init_on_alloc=1`), Freelist Shuffle (`page_alloc.shuffle=1`), Kstack Offset Randomization | Optimized Fast Path (Mitigations tuned for peak compute throughput) | Standard Enterprise Hardening (Compatibility & certification focused) |\n| **Attack Surface Truncation** | Explicit Blacklisting + `/bin/false` stubbing of 16+ legacy network protocols & filesystems | Dynamic Module Autoloading (Broad hardware/network interoperability) | Dynamic Module Autoloading (Signed RPM enterprise drivers) |\n\n---\n\n## 1. Kernel Architecture, Toolchains & Compilation Strategy\n\n### 1.1 Microarchitecture ISA Baseline (x86-64-v3/v4 vs Generic Baseline)\n* **User System (NixOS)**: Standard `nixpkgs` binaries are compiled against generic `x86-64` (v1/v2 baseline with selective runtime dispatch) to preserve broad binary compatibility across the ecosystem. Unless explicitly rebuilding packages with `stdenv.hostPlatform.gcc.arch = \"x86-64-v3\"`, modern CPU instructions (AVX2, FMA3, BMI2) remain unexploited in non-glibc computational hot loops.\n* **CachyOS**: Re-architects the distribution around multi-tier repositories (`x86-64-v3` requiring AVX, AVX2, BMI1, BMI2, F16C, FMA, MOVBE; and `x86-64-v4` requiring AVX-512F/CD/BW/DQ/VL). Instruction scheduling, register allocation (16 vs 32 vector registers on AVX-512), and auto-vectorization across the entire userspace and kernel deliver a **5%\u201320% IPC uplift** in cryptography, compression/decompression, and string routines without runtime CPU dispatch overhead.\n* **Fedora**: Enforces `x86-64-v2` baseline (SSE4.2, POPCNT, CMPXCHG16B). While Fedora leverages `glibc-hwcaps` for dynamic loading of optimized shared libraries on modern hardware, the base kernel and standard packages remain constrained to avoid breaking legacy virtualization instances and older hardware.\n\n### 1.2 Toolchain, Compiler Optimization & Link-Time Optimization (LTO)\n* **CachyOS**: Employs an aggressive toolchain pipeline:\n  * **Kernel**: Built using LLVM/Clang with `CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE_O3=y`, `CONFIG_LTO_CLANG_THIN=y` (or Full LTO), and AutoFDO/BOLT (Binary Optimization and Layout Tool) profiling. This eliminates cross-translation-unit call overhead, inlines critical fast paths (page allocator locks, spinlocks, VFS lookups), and optimizes branch layout to minimize instruction cache (iCache) misses.\n  * **Userspace**: Built with `-O3 -march=x86-64-v3 -flto=auto`.\n* **User System (NixOS)**: Built with upstream standard GCC `-O2`. Standard Linux kernel configurations do not enable Clang LTO by default. GCC produces highly stable, predictable binaries with fast compilation times, but lacks cross-object inlining across the kernel core and dynamic loadable modules (`.ko`).\n* **Fedora**: Standardized on GCC `-O2`. Fedora rejects kernel LTO to maintain broad DKMS module compatibility, rapid distribution build farm throughput, and deterministic debugging symbols (`DWARF5`). It prioritizes compiler-level security hardening (`-fstack-protector-strong`, `-fstack-clash-protection`, Control Flow Integrity/IBT, and `-D_FORTIFY_SOURCE=3`).\n\n### 1.3 Preemption Models, Tick Rate (HZ) & RCU Subsystem\n* **User System (`boot.nix`)**:\n  * Employs `CONFIG_PREEMPT_DYNAMIC=y` on Linux 7.2.0.\n  * Explicitly passes `rcutree.enable_rcu_lazy=1` in `boot.kernelParams`.\n  * **Technical Impact**: `rcu_lazy` batches RCU callbacks (especially for memory reclamation and file closures) on idle or power-constrained CPUs for up to several seconds instead of waking up cores immediately. This significantly improves energy efficiency and reduces idle context switching on battery power at the cost of slightly higher peak memory retention during high churn.\n* **CachyOS**: Defaults to `CONFIG_PREEMPT_DYNAMIC=y` configured for full dynamic preemption, paired with `HZ=500` or `HZ=1000`. CachyOS applies kernel patches to reduce scheduler timer jitter, minimize timer wheel lock contention, and enable low-latency task preemption for interactive audio and rendering pipelines.\n* **Fedora**:\n  * **Fedora Server**: Configured for high-throughput batch execution (`CONFIG_PREEMPT_VOLUNTARY` or dynamic preemption set to `voluntary`/`none`) with low timer tick rates (`HZ=250` or `HZ=100` on server flavors). This minimizes timer interrupt overhead, preserves hardware CPU cache lines, and maximizes throughput for network I/O, hypervisors (KVM), and containerized execution.\n  * **Fedora Workstation**: Uses `HZ=1000` with `CONFIG_PREEMPT_DYNAMIC=y`.\n\n---\n\n## 2. CPU Scheduling, Task Prioritization & Power Governance\n\n### 2.1 Kernel CPU Schedulers: EEVDF vs BORE vs sched_ext\n* **User System (NixOS)**:\n  * Runs standard upstream **EEVDF** (Earliest Eligible Virtual Deadline First).\n  * **Mechanics**: Replaced CFS in Linux 6.6+. EEVDF tracks `vruntime` (virtual runtime spent on CPU) and `lag` (difference between runtime allocation entitlement and actual consumption). Tasks calculate an eligible time and a virtual deadline:\n    $$\\text{Deadline} = \\text{Eligible Time} + \\frac{\\text{Time Slice}}{\\text{Weight}}$$\n  * **Trade-off**: High fairness, zero starvation, mathematically robust latency bounds. However, when large batch tasks (e.g., `nix-build`, `cargo build`) saturate all hardware threads, short interactive bursts can still experience micro-stalls if their virtual deadlines align with saturated compute tasks.\n* **CachyOS**:\n  * **BORE (Burst-Oriented Response Enhancer)**: Evaluates task burstiness by calculating:\n    $$\\text{Burst Score} = \\frac{\\text{CPU Burst Time}}{\\text{Total Sleep Time} + \\text{CPU Execution Time}}$$\n    Tasks with intermittent execution (GUI event loops, audio threads, input handlers, frame dispatchers) receive minimal burst scores. BORE scales their time-slice granularity down and shifts their virtual deadlines earlier, rendering the interactive desktop completely immune to frame drops even under 100% synthetic CPU load.\n  * **sched_ext (SCX)**: CachyOS includes full out-of-tree/upstream support for eBPF schedulers (`scx_rusty`, `scx_lavd`, `scx_bpfland`, `scx_flash`). This enables swapping scheduling algorithms live in production without kernel rebuilds\u2014e.g., using `scx_lavd` for gaming-optimized core affinity or `scx_rusty` for multi-socket NUMA load balancing.\n* **Fedora Server/Base**:\n  * Strictly upstream **EEVDF**. Prioritizes standard CFS replacement without out-of-tree patches. Excellent for cloud containers, virtualization, and uniform database execution where algorithmic determinism and upstream kernel debuggability are paramount.\n\n### 2.2 Task Priority & Latency Management: ananicy-cpp\n* **User System (`power.nix`)**:\n  * Deploys `ananicy-cpp` with `ananicy-rules-cachyos`.\n  * Configured with `cgroup_load = false`, `apply_cgroup = false`, and `cgroup_realtime_workaround = false`.\n  * **Mechanics**: Runs as a lightweight C++ daemon monitoring `procfs`/`netlink` process lifecycle events. Upon matching process rule definitions, it applies:\n    1. POSIX `nice` levels (-20 to +19).\n    2. Real-time scheduling policies (`SCHED_FIFO`, `SCHED_RR`, `SCHED_IDLE`, `SCHED_BATCH`).\n    3. I/O scheduling classes (`ionice -c 1` Realtime, `-c 2` Best Effort, `-c 3` Idle).\n    4. `sched_latency_nice` / `sched_idle` hints.\n  * **Operational Caveat**: Because `apply_cgroup = false`, priority management is purely per-PID rather than enforced across hierarchical cgroup v2 controller boundaries (`cpu.weight`, `io.weight`). This avoids cgroup nesting conflicts but loses kernel-enforced aggregate slice throttling.\n* **CachyOS**: Ships `ananicy-cpp` pre-integrated with systemd cgroup hierarchy and CachyOS kernel-specific latency-nice patches, dynamically tuning both PID-level scheduler attributes and cgroup group weights.\n* **Fedora**: Completely omits dynamic auto-renicing daemons. Relies entirely on static systemd resource partitioning (`user.slice`, `system.slice`, `machine.slice`), standard `rtkit` (RealtimeKit) for audio/D-Bus privilege escalation, and unmanaged EEVDF fair sharing.\n\n### 2.3 Frequency Scaling & Power Management: auto-cpufreq vs PPD vs tuned\n* **User System (`power.nix` & `power-server.nix`)**:\n  * Laptop configuration deploys `auto-cpufreq` paired with `thermald`:\n    * **AC**: `governor = \"powersave\"`, `energy_performance_preference = \"balance_performance\"`, `turbo = \"auto\"`.\n    * **Battery**: `governor = \"powersave\"`, `energy_performance_preference = \"power\"`, `turbo = \"auto\"`, battery charge thresholds 20%\u201380%.\n    * **Server configuration**: Static `powerManagement.cpuFreqGovernor = \"powersave\"`.\n  * **Mechanics**: `auto-cpufreq` actively monitors CPU usage, temperatures, and AC/battery state via userspace polling, overriding sysfs attributes under `/sys/devices/system/cpu/cpu*/cpufreq/`.\n  * **Trade-off**: Excellent automated power saving on laptops, but can introduce governor-switching latency spikes or conflict with hardware-autonomous P-state management on modern AMD (`amd-pstate-epp`) and Intel (`intel_pstate`) CPUs.\n* **CachyOS**: Relies primarily on kernel-native drivers (`amd-pstate=active`, `intel_pstate=active`) coupled with `power-profiles-daemon` or custom sysctl profiles. Avoids continuous userspace polling loops, letting hardware autonomous energy performance hints (EPP register writes) handle microsecond-level frequency transitions.\n* **Fedora**:\n  * **Fedora Workstation**: Employs `power-profiles-daemon`, directly wired into GNOME/KDE settings and ACPI `platform_profile` sysfs interfaces.\n  * **Fedora Server**: Deploys `tuned` (`tuned-adm`), which applies holistic, workload-specific tuning profiles (e.g., `throughput-performance`, `latency-performance`, `network-throughput`, `virtual-guest`). `tuned` adjusts scheduler parameters, disk queue schedulers, PCIe power states, sysctl VM variables, and CPU EPP settings in an integrated daemon.\n\n---\n\n## 3. Memory Subsystem, ZRAM, VM Paging & OOM Management\n\n### 3.1 ZRAM Architecture, Swap Mechanics & Compression Algorithms\n* **User System Configuration**:\n  * `zramSwap.enable = true`, `algorithm = \"zstd\"`, `memoryPercent = 100`, `memoryMax = 8GB`, `priority = 100`.\n  * Explicitly disables `zswap` in `systemd.tmpfiles.rules`:\n    `w /sys/module/zswap/parameters/enabled - - - - 0`\n* **Architectural Breakdown**:\n  * **ZRAM vs zswap**: ZRAM is an in-memory compressed block device with its own swap signature. `zswap` is a front-end cache for a real backing swap disk. Forcing `zswap=0` prevents redundant dual-compression layers and memory thrashing.\n  * **zstd vs lz4 / lzo-rle**:\n    * `zstd` achieves high compression ratios (**2.6:1 to 3.2:1** for mixed memory/workload strings). An 8GB ZRAM partition effectively holds ~20GB\u201325GB of uncompressed anonymous pages at the cost of slight decompression CPU overhead.\n    * `lzo-rle` (Fedora default) achieves lower compression (**1.8:1 to 2.2:1**), but decompresses at multi-gigabyte-per-second rates per core with minimal CPU cycle overhead.\n  * **Priority 100 / 32767**: Placing ZRAM swap priority higher than physical disk swap (if any) ensures the kernel drains memory into compressed RAM before ever hitting NVMe/SATA storage.\n\n### 3.2 Deep Dive into VM Sysctl Parameters\n\n#### A. `vm.swappiness` (150 / 180 vs 100 vs 60)\n* **Kernel Mechanism**: In `mm/vmscan.c`, the kernel calculates the ratio of anonymous scan priority to file cache scan priority:\n  $$\\text{Anon Priority} = \\text{swappiness}$$\n  $$\\text{File Priority} = 200 - \\text{swappiness}$$\n* **Analysis**:\n  * **Fedora Server (`swappiness=60`)**: Prioritizes evicting file-backed page caches (VFS, shared objects, executable pages) over anonymous memory. When file pages are dropped, subsequent accesses require re-reading from disk (blocking NVMe/SSD I/O stalls).\n  * **User System (`swappiness=150` on Laptop, `180` on Server) & CachyOS (`swappiness=150`)**: Swapping to ZRAM is pure RAM-to-RAM compression, operating at memory-bus speeds without disk queue overhead. A swappiness of 150\u2013180 forces the kernel to compress cold anonymous heap pages into ZRAM while aggressively protecting active file caches, eliminating storage read latency during heavy compilation or application switching.\n\n#### B. `vm.page-cluster` (0 vs 3)\n* **Kernel Mechanism**: Controls sequential page readahead during swap-in operations. The number of pages read simultaneously is $2^{\\text{page-cluster}}$.\n* **Analysis**:\n  * `page-cluster = 3` (Upstream default): Reads $2^3 = 8$ pages (32KB). On mechanical HDDs, this amortizes head seek latency.\n  * `page-cluster = 0` (User System & CachyOS): Reads $2^0 = 1$ page (4KB). Because ZRAM is purely random-access RAM with algorithmic decompression, fetching 8 adjacent pages causes massive read amplification and wastes CPU cycles decompressing unused pages. Setting this to `0` reduces swap-in latency to the absolute minimum.\n\n#### C. `vm.vfs_cache_pressure` (50 vs 100)\n* **Kernel Mechanism**: Governs the kernel's tendency to reclaim directory entries (`dentries`) and inode caches relative to standard page cache.\n* **Analysis**:\n  * `vfs_cache_pressure = 100` (Fedora): Standard fair reclamation.\n  * `vfs_cache_pressure = 50` (User System & CachyOS): Halves the rate of dentry and inode eviction. Keeps file tree metadata in RAM, dramatically accelerating git operations, recursive file searches (`find`, `fd`, `ripgrep`), and multi-file code builds (`nix-build`, `cargo`, `gcc`) without hitting disk metadata tables.\n\n#### D. `vm.watermark_boost_factor` & `vm.watermark_scale_factor`\n* **Kernel Mechanism**:\n  * `watermark_scale_factor` defines the gap between `WMARK_MIN`, `WMARK_LOW`, and `WMARK_HIGH` as a percentage of zone size ($\\text{gap} = \\frac{\\text{scale\\_factor}}{10000} \\times \\text{Zone Size}$).\n  * `watermark_boost_factor` inflates memory watermarks whenever high-order memory allocations fail, attempting proactive compaction.\n* **Analysis**:\n  * **User System (`boot.nix`) & CachyOS**:\n    * `watermark_boost_factor = 0`: Disables dynamic watermark inflation. Eliminates memory compaction micro-stutters and sudden CPU spikes during gaming, audio rendering, and interactive desktop use.\n    * `watermark_scale_factor = 125` (1.25% of zone size vs standard 0.1%): Widens the buffer between `WMARK_MIN` and `WMARK_LOW`. `kswapd` wakes up early and reclaims memory smoothly in the background, preventing processes from dipping below `WMARK_MIN` and hitting blocking **direct reclaim** (`direct_reclaim`).\n  * **Fedora**: Retains upstream defaults (`boost_factor=15000`, `scale_factor=10`), which preserves maximum raw allocatable memory for bulk server workloads but allows direct-reclaim stutter during sudden memory allocation bursts.\n\n#### E. Page Cache Dirty Memory Throttling: Absolute Bytes vs Percentages\n* **Kernel Mechanism**:\n  * `vm.dirty_background_bytes` / `dirty_background_ratio`: Threshold where kernel `flusher` threads begin asynchronous disk writeback.\n  * `vm.dirty_bytes` / `dirty_ratio`: Hard threshold where application write calls block synchronously until dirty pages drop below the ceiling.\n* **Analysis**:\n  * **Fedora (`dirty_ratio=20%`, `dirty_background_ratio=10%`)**: On a 32GB\u2013128GB RAM system, 20% dirty ratio allows **6.4GB to 25.6GB** of unwritten data to buffer in RAM. When dirty sync flushes occur, or when slow USB/SATA drives are written to, writeback queues fill up and cause total system lockups (\"desktop freeze on large write\").\n  * **User System (`boot.nix`) & CachyOS**:\n    * `dirty_background_bytes = 67108864` (64MB).\n    * `dirty_bytes = 268435456` (256MB).\n    * `dirty_writeback_centisecs = 1500` (15s), `dirty_expire_centisecs = 1500` (15s).\n    * **Technical Impact**: Hard caps dirty data to 256MB regardless of total RAM size. Asynchronous disk flushing starts at 64MB, ensuring steady, non-blocking I/O throughput and completely preventing writeback stalls on storage devices.\n\n### 3.3 Transparent Hugepages (THP) & khugepaged Optimization\n* **User System (`cachyos-tuning.nix`)**:\n  * `transparent_hugepage/defrag = defer+madvise`\n  * `transparent_hugepage/khugepaged/max_ptes_none = 409`\n* **Mechanics & Technical Impact**:\n  * **`defrag = defer+madvise`**: In standard `always` or direct `madvise` defrag mode, an application requesting a 2MB hugepage blocks execution until the kernel compacts fragmented memory to construct a contiguous 2MB frame. With `defer+madvise`, if a 2MB page is not instantly available in the free list, the kernel falls back to standard 4KB pages immediately without blocking the application, queuing asynchronous background promotion via `khugepaged`.\n  * **`khugepaged/max_ptes_none = 409`**: Controls how many empty 4KB page slots (out of 512 in a 2MB range) `khugepaged` will tolerate when collapsing fragmented pages into a hugepage. The default is 511 (which collapses sparse memory, causing massive RSS memory bloat). Setting `409` requires at least $\\ge 20\\%$ page occupancy ($512 - 409 = 103$ active pages) before triggering hugepage allocation, preventing memory amplification.\n\n### 3.4 Out-of-Memory Handling: systemd-oomd vs earlyoom\n* **User System (`power.nix`, `power-server.nix`)**:\n  * `systemd.oomd.enable = true` with `enableUserSlices = true; enableSystemSlice = false;`.\n  * `sshd.serviceConfig.OOMScoreAdjust = -1000` (protects SSH session from termination).\n* **Mechanics**:\n  * `systemd-oomd` operates by monitoring kernel **Pressure Stall Information (PSI)** (`/proc/pressure/memory` and `io`). When memory pressure exceeds defined duration thresholds (e.g., 20s cumulative stall > 50%), `systemd-oomd` acts proactively by killing the most memory-intensive sub-cgroup within `user.slice`.\n  * Because `enableSystemSlice = false`, critical system infrastructure daemons are never killed.\n* **CachyOS**: Integrates `systemd-oomd` or `earlyoom` with tighter PSI polling loops, dynamically terminating runaway browser tabs or build tasks before the kernel enters hard page thrashing.\n* **Fedora**: Standardized on `systemd-oomd`. While effective at preventing hard kernel lockups, Fedora's default configuration on workstations has historically faced criticism for abruptly terminating large desktop applications or compiler processes before swapping had fully exhausted available memory.\n\n---\n\n## 4. Storage, Filesystems & I/O Architecture\n\n### 4.1 Root & State Lifecycle: Ephemeral Impermanence vs. Stateful Mutation\n\n#### User System (Impermanence Model)\n* **Laptop Architecture (`tmpfs` on `/`)**:\n  - The root filesystem (`/`) exists strictly in RAM as a volatile `tmpfs` (bounded to `25%` of physical RAM).\n  - Mutable persistent state is explicitly isolated to a dedicated Btrfs subvolume (`/@persistent`) and bound to system target paths via the NixOS `preservation` module (`/etc/machine-id`, `/var/log/journal`, `/etc/NetworkManager`, `/var/lib/tailscale`, etc.).\n  - **Mechanisms & Guarantees**: Any unauthorized runtime mutation, malware drop in `/tmp`, `/var`, `/usr`, or temporary drift is annihilated upon reboot. It enforces a **mathematically deterministic root state** derived entirely from the Nix flake.\n* **Server Architecture (Btrfs Snapshot Rollback in Initrd)**:\n  - An initrd systemd service (`initrd.systemd.services.rollback-root`) executes prior to mounting `sysroot`:\n    ```bash\n    mount -t btrfs -o subvol=/ /dev/sda /mnt\n    btrfs subvolume delete /mnt/@\n    btrfs subvolume snapshot /mnt/@blank /mnt/@\n    umount /mnt\n    ```\n  - **Mechanisms & Guarantees**: Eliminates state drift on disk without requiring all root writes to fit within RAM. Storage overhead is zero-copy (COW metadata pointer reassignment).\n\n#### CachyOS (Server/Base)\n* Utilizes a standard persistent layout (typically Btrfs subvolumes `@` and `@home`, or XFS/Ext4).\n* Filesystem mutations accumulate over time. While Btrfs snapshots can be triggered via `snapper` or `btrfs-assistant`, rollback is an administrative recovery action rather than an intrinsic per-boot lifecycle invariant.\n* State accumulation allows configuration drift, orphan packages, and persistent runtime artifacts in `/var` and `/etc`.\n\n#### Fedora Server\n* Defaults to an **LVM volume group containing XFS partitions** (or Stratis for pooled storage abstraction).\n* Root `/` and data mounts are stateful, rigid block allocations. XFS does not support snapshot rollback to a clean state without external LVM thin-pool snapshots.\n* State management relies entirely on `dnf`/`rpm-ostree` transaction integrity.\n\n---\n\n### 4.2 Storage Engines: Mount Parameters & Block Allocators\n\n#### Detailed Breakdown:\n\n1. **User's Btrfs Configuration (`compress=zstd:1, noatime, discard=async, space_cache=v2`)**:\n   - `compress=zstd:1`: Uses level-1 Zstandard compression in the kernel VFS pipeline. This minimizes CPU cycles during writeback while providing near-optimal compression ratios for system binaries, source trees, and text files. It reduces write amplification on flash memory.\n   - `noatime`: Completely disables inode access time writes upon file read operations, eliminating a write I/O transaction for every read.\n   - `discard=async`: Replaces blocking synchronous discards with an asynchronous background block-group trimming queue. Avoids I/O stalls during file deletion operations on NVMe/SATA SSDs.\n   - `space_cache=v2`: Free space tree stored as an indexed on-disk B-tree, preventing severe performance degradation on large filesystems compared to v1 bitmap caching.\n\n2. **User's ZFS Mirrored Data Pool (`ashift=12, recordsize=1M, compression=zstd, posixacl, xattr=sa, dnodesize=auto, atime=off`)**:\n   - `ashift=12`: Aligns pool vdev block allocation to $2^{12} = 4096$ bytes, matching the physical 4Kn/512e Advanced Format sector size of the underlying mechanical drives. Completely avoids read-modify-write performance penalties.\n   - `recordsize=1M` (on `/mnt/data/media`): Overrides the default 128KB recordsize for bulk sequential media workloads. This reduces ZFS metadata overhead by 8x, increases sequential read/write throughput, and improves compression efficiency on large data streams.\n   - `xattr=sa`: Stores extended attributes directly inside the inode dnode (when space permits) rather than creating an auxiliary hidden directory and separate disk block allocation. This drops file attribute lookups from 2-3 I/O seeks to 1 seek.\n   - `atime=off`: Prevents continuous metadata write cycles to the mirrored spinning platters on read hits.\n   - `zfs_arc_min=512M`, `zfs_arc_max=2G`: Strictly bounds the Adaptive Replacement Cache in kernel space to prevent starving user-space processes on memory-constrained server nodes.\n\n3. **Fedora Server Engine (XFS & LVM)**:\n   - **XFS Internals**: Allocation Groups (AGs) divide the block device into independent sub-regions, allowing massive concurrent parallel allocation across multi-core systems. Reflink copy-on-write (`reflink=1`) is enabled by default.\n   - **Limitations**: XFS has no native data checksumming (only metadata CRC protection), cannot perform live inline transparent compression, and cannot self-heal bitrot on mirrored disks.\n\n4. **CachyOS Engine**:\n   - Utilizes Btrfs or XFS with kernel-level performance patches. CachyOS packages native OpenZFS builds synchronized with their custom kernel updates (avoiding DKMS compile latency), but does not ship fine-grained per-dataset tuning out of the box.\n\n---\n\n### 4.3 Block I/O Queuing: blk-mq Schedulers & Latency Characteristics\n* **NVMe (`none`)**:\n  - Direct hardware submission via multi-queue block layer (`blk-mq`). Modern NVMe controllers expose dozens of hardware queues matching CPU core counts.\n  - Bypassing the software I/O scheduler eliminates lock contention, CPU cache misses, and scheduling latency, maximizing IOPS (400k+ IOPS) at sub-millisecond tail latencies.\n* **SATA SSD (`mq-deadline`)**:\n  - SATA SSDs feature a single hardware queue (NCQ depth 32). `mq-deadline` enforces hard expiration deadlines on read (500ms) and write (5s) requests while grouping requests into contiguous block sectors.\n  - Prevents read starvation caused by heavy background write streams.\n* **Rotational Mechanical Disks (`bfq` - Budget Fair Queueing)**:\n  - Rotational disks suffer from head seek latency (~10ms per seek). `bfq` assigns per-process disk budgets and tracks disk seek time, multiplexing requests to maximize rotational throughput while preventing background processes (like ZFS scrubs or backups) from freezing interactive applications.\n\n---\n\n## 5. Network Stack & Socket Tuning\n\n### 5.1 Congestion Control & Qdisc Architecture\n* **User System: TCP BBR + FQ Qdisc**:\n  * **Mechanism**: BBR (Bottleneck Bandwidth and Round-trip propagation time) does not use packet loss as an indicator of congestion (unlike legacy Reno/CUBIC). Instead, it constructs a real-time model of the physical network path by measuring maximum delivery rate ($BtlBw$) and minimum round-trip time ($RTprop$).\n  * **Qdisc Coupling**: BBR **requires** the `fq` (Fair Queueing) qdisc (`net.core.default_qdisc = \"fq\"`) because it relies on FQ's per-flow pacing timers to inject packets into the wire smoothly at the estimated bottleneck rate, completely preventing packet bursts that cause bufferbloat in intermediate routers.\n  * **Characteristics**: Maximum throughput over lossy/long-distance WAN links; immune to shallow-buffer packet drop penalties.\n* **CachyOS: TCP BBRv3 + CAKE / FQ**:\n  * **Mechanism**: CachyOS incorporates upstream developmental **BBRv3** kernel patches. BBRv3 refines the pacing gain and loss response algorithms, addressing BBRv1's aggressive behavior towards CUBIC flows and improving dynamic convergence when multiple flows share a bottleneck buffer.\n  * **CAKE Integration**: CAKE (Common Applications Kept Enhanced) provides multi-tier priority queueing (DiffServ), active queue management (AQM), and automated per-host fairness.\n* **Fedora: TCP CUBIC / BBR + FQ-CoDel**:\n  * **Mechanism**: Fedora defaults to upstream Linux conventions (`CUBIC` + `fq_codel`).\n  * **Characteristics**: CUBIC is a window-based congestion control algorithm where the congestion window is a cubic function of time since the last loss event. On high-speed, long-distance WAN connections with minimal packet loss, CUBIC can prematurely slash throughput by 30-50% upon a single dropped packet. `fq_codel` mitigates bufferbloat well on local interfaces but does not pace TCP packets at the socket level like `fq`.\n\n### 5.2 Socket Buffer & Sysctl Deep-Dive\n* **`net.ipv4.tcp_fastopen = 3`**: Enables TCP Fast Open for both incoming (listen) and outgoing (connect) sockets. Clients send application data (e.g., HTTP GET or TLS ClientHello) directly inside the initial SYN packet. Eliminates one entire Round-Trip Time ($1 \\times \\text{RTT}$) from connection handshakes.\n* **`net.ipv4.tcp_mtu_probing = 1`**: Enables Path MTU Discovery via probing (RFC 4821). If an intermediate router drops ICMP \"Fragmentation Needed\" packets (creating an MTU blackhole), the kernel automatically detects the lack of ACKs on full-sized segments and dynamically reduces the MSS rather than hanging the connection indefinitely.\n* **`net.ipv4.tcp_rfc1337 = 1`**: Enforces strict TCP TIME_WAIT state protection. Disallows incoming RST packets from prematurely killing a socket in `TIME_WAIT` state, protecting services against connection hijacking and corrupted data delivery from delayed packets.\n* **`net.ipv4.tcp_syncookies = 1`**: Cryptographic SYN-Queue fallback during SYN flood attacks.\n* **`net.core.netdev_max_backlog = 4096` (Laptop) / `10000` (Server)**: Expands device input ring buffers to prevent packet drops during bursty network arrival.\n\n---\n\n## 6. Security, Hardening & Attack Surface\n\n### 6.1 Linux Security Modules (LSM) & Access Control Paradigms\n* **User System: Landlock + Yama + Hardened BPF**:\n  * **Landlock LSM**: Allows unprivileged user processes (browsers, network servers) to restrict their own access to the VFS hierarchy at runtime via programmatic system calls without requiring root privileges or system-wide policy recompilation.\n  * **Yama (`ptrace_scope = 1`)**: Restricts process memory inspection via `ptrace()`. A process can only attach to its direct child processes. Prevents unprivileged processes from injecting shellcode into running sibling processes (e.g., extracting SSH keys or credentials from other user processes).\n  * **eBPF Hardening**: `unprivileged_bpf_disabled = 1` prevents non-root users from compiling and attaching eBPF programs to the kernel. `net.core.bpf_jit_harden = 2` blinds JIT compiler constants against speculative execution and ROP gadget synthesis.\n* **Fedora: SELinux Enforcing (`targeted`)**:\n  * **Mechanism**: Full Mandatory Access Control (MAC) based on Flask architecture. Every process (subject) and file/socket/node (object) is assigned a security context (`user:role:type:level`).\n  * **Guarantees**: Provides strict confinement for system daemons and root containers (`svirt_lxc_net_t`). Even if a daemon is compromised as UID 0 (root), it cannot read unauthorized files outside its explicit policy domain.\n  * **Trade-offs**: Significant operational complexity, label relabeling overhead (`fixfiles`), and CPU overhead during heavy VFS traversal.\n* **CachyOS: DAC Baseline**:\n  * Primarily relies on standard POSIX Discretionary Access Control (file permissions, capabilities, and systemd sandboxing).\n  * Minimal MAC enforcement out of the box to eliminate lock contention in low-latency gaming and compilation workloads.\n\n### 6.2 Kernel Hardening Sysctls & Memory Allocator Mitigations\n* **`init_on_alloc=1`**: The kernel fills allocated page frames and slab objects with zeroes prior to handing them to kernel subsystems or user space. This mitigates entire classes of **Use-Before-Initialization** vulnerabilities and memory content disclosure across security domains.\n* **`page_alloc.shuffle=1`**: Randomizes the page buddy allocator's free list order. It disrupts deterministic heap feng-shui and exploitation techniques that rely on predictable page positioning in physical memory.\n* **`randomize_kstack_offset=on`**: Adds a random entropy offset to the kernel stack frame on every single system call invocation. Attackers cannot rely on deterministic stack offsets when attempting kernel Return-Oriented Programming (ROP) or stack-smashing exploits.\n* **`kernel.kptr_restrict = 2`**: Completely hides kernel pointers from `/proc` files even for root users.\n* **`kernel.dmesg_restrict = 1`**: Restricts dmesg access to `CAP_SYSLOG` (prevents information leaks).\n* **`fs.protected_fifos = 2` & `fs.protected_regular = 2`**: Prevents FIFO/file spoofing in world-writable sticky directories (`/tmp`).\n\n### 6.3 Attack Surface Reduction: Module Blacklisting & Stubs\nThe User's configuration actively truncates legacy, unmaintained, and exotic kernel code paths.\n\n#### 1. Blacklisted Protocol Modules:\n* `dccp` (Datagram Congestion Control Protocol)\n* `sctp` (Stream Control Transmission Protocol)\n* `rds` (Reliable Datagram Sockets)\n* `tipc` (Transparent Inter-Process Communication)\n* `ax25`, `netrom`, `rose` (Amateur radio packet protocols)\n* `atm` (Asynchronous Transfer Mode)\n* `rxrpc`, `esp4`, `esp6`\n\n#### 2. Blacklisted Legacy Filesystems:\n* `cramfs`, `freevxfs`, `jffs2`, `hfs`, `hfsplus`\n\n#### 3. Defensive Enforcement Mechanism:\nIn addition to standard blacklisting, the user enforces **Modprobe Execution Stubs**:\n```nix\ninstall  /nix/store/.../bin/false\n```\n* **Why standard `blacklist` is insufficient**: Standard modprobe blacklisting only prevents module autoloading based on hardware alias detection. If an unprivileged application invokes `socket(AF_NETROM, ...)` or attempts to mount a malicious USB key containing an HFS filesystem, the kernel's `request_module()` will bypass the blacklist and dynamically load the module anyway.\n* **The `/bin/false` Stub Effect**: The stub intercepts the kernel's dynamic load request and immediately returns a failure code (`1`), completely neutralizing the attack surface of legacy codebases known for recurring privilege escalation vulnerabilities (e.g., CVE-2021-43267 in TIPC, historical RDS zero-days).\n\n---\n\n## 7. Architectural Conclusions & Recommendations\n\n1. **User System (NixOS Impermanence + Hardened Custom Kernel)**:\n   - **Verdict**: A **Zero-Trust, High-Determinism Architecture**.\n   - **Strengths**: Perfect state reproducibility, complete immunity to filesystem bitrot and configuration drift, maximized WAN TCP throughput via BBR pacing, and proactive kernel memory safety mitigations.\n   - **Remaining Optimization Potential**: Can bridge the compiler gap with CachyOS by building performance-critical binaries/kernel with `stdenv.hostPlatform.gcc.arch = \"x86-64-v3\"` or leveraging Clang `-O3` + ThinLTO overlays.\n2. **CachyOS (Server/Base)**:\n   - **Verdict**: An **Ultra-Low Latency & High-Throughput Compute Platform**.\n   - **Strengths**: Leads in single-thread IPC, instruction vectorization (`x86-64-v3/v4`), dynamic CPU scheduling (BORE/SCX), and bleeding-edge networking (BBRv3/CAKE).\n   - **Trade-offs**: Sacrifices deterministic state management, kernel exploit mitigations (`init_on_alloc`), and mandatory access control.\n3. **Fedora (Server/Base)**:\n   - **Verdict**: An **Enterprise Compliance & Managed Isolation Platform**.\n   - **Strengths**: Mature SELinux confinement, Red Hat ecosystem certification, and high-concurrency multi-threaded I/O via XFS Allocation Groups.\n   - **Trade-offs**: Relies on legacy state accumulation, conservative TCP defaults (CUBIC/fq_codel), and lacks built-in bitrot-resilient storage primitives.\n", "creation_timestamp": "2026-08-23T11:14:42.889824Z"}