What should I monitor/log and how should I monitor/log to determine why my headless NAS is often becoming unavailable?

The problem:

• Another machine that depends on the NAS routinely has its services unavailable because the NFS mounts are no longer mounted.
• When that happens, sometimes a sudo mount -a recovers them.
• Other times, the NAS is not pingable, so I go to the physical host, plug in monitor/keyboard and find that I can’t log in. The login screen is frozen, requiring hard reboot.
• Often when I leave a monitor attached (VGA), I come back to a screen that says:

critical medium error, dev sda, sector 163776752 op 0x0:(READ) flags 0x700 phys_seg 1 prio class 2

I started a sudo smartctl -t long /dev/sda a few hours ago, and sometime since then, the server depending upon it no longer had NFS mounted. But a simple sudo mount -a resolved.

What the server was also doing when it had a network blip:

• rclone was backing up to backblaze b2
• Acting as NFS server for Plex/*arr media server
• Acting as NFS storage for Proxmox machine (but no VMs or CTs running)

Pasted some zpool output below. Details about the machine:

• Repurposed old hardware, just built this Debian 12 NAS a couple months ago

• Operates as backup destination for other machines

• Operates as media location for my Plex machine - other server that mounts the NAS via NFS.

• P6X58D-E LGA 1366 motherboard, Intel X5670 CPU, 18 GB (3x4GB, 3x2GB triple channel)

• 8 hard drives connected to LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)

• 10GbE to managed TP-Link switch through one port on Mellanox Connectx-3 MCX312A-XCBT EN

  ➜ sudo zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT nvr 5.45T 3.35T 2.10T - - 2% 61% 1.00x ONLINE - tank 70.9T 34.4T 36.5T - - 0% 48% 1.00x ONLINE -

  ➜ sudo zpool status -v pool: nvr state: ONLINE scan: scrub repaired 0B in 08:49:40 with 0 errors on Sun Nov 12 09:13:41 2023 config:

        NAME            STATE     READ WRITE CKSUM
        nvr             ONLINE       0     0     0
          mirror-0      ONLINE       0     0     0
            6T-75LN0J4  ONLINE       0     0     0
            6T-95A2PNV  ONLINE       0     0     0
  

  errors: No known data errors

  pool: tank
  

  state: ONLINE scan: scrub repaired 1M in 16:44:16 with 0 errors on Sun Nov 12 17:08:27 2023 config:

        NAME              STATE     READ WRITE CKSUM
        tank              ONLINE       0     0     0
          raidz1-0        ONLINE       0     0     0
            12T-5PGJ4A0D  ONLINE       0     0     0
            12T-Z2J26EBT  ONLINE       0     0     0
            12T-5PGHSZJC  ONLINE       0     0     0
          raidz1-1        ONLINE       0     0     0
            14T-9KG38U5L  ONLINE       0     0     0
            14T-9KG81HRL  ONLINE       0     0     0
            14T-9RGG5ZDC  ONLINE       0     0     0
  

  errors: No known data errors