System freeze

My backup Kubuntu workstation began freezing shortly after boot recently. The problem was recurrent for a week or so and then I just switched the damn thing off. Have picked this up again recently to try and fix, because I would like to make use of this machine’s disks.

I ran MemTest86, and that surfaced no errors. Seems as though the RAM is okay and probably the CPU too.

The first thing I tried after the memory test was to remove the WD Blue 250GB SATA SSD M.2 which was in use as a ZFS cache. And this seems to have fixed the problem! Can’t be sure yet, but so far so good. Will continue to keep an eye on things…

Here are some photos for those playing along at home:

Replace a disk in a ZFS pool

So smartd is suddenly emailing me about problems with one of the disks in my ZFS zpool. I have ordered a replacement disk and am waiting for it to arrive. While the smartd email says there is a problem `zpool status` says everything is fine. So I’m running a `zpool scrub` to see if ZFS can pick up on the disk errors.

Preparing for the disk replacement I searched the web and found Replace a disk in a ZFS pool.

I found the serial number of the faulty disk with `lsblk -I 8 -d -o NAME,SIZE,SERIAL`. The process is then:

  1. Shutdown the server
  2. Replace the faulty disk
  3. Boot the server
  4. Run zpool replace: sudo zpool replace data sdc
  5. Check zpool status: sudo zpool status data

I hope it turns out to be that easy! Now I just wait for my scrub to complete and my disk to arrive.

Update

Click through on the link below for some excellent documentation about how to handle this error:

Every 2.0s: zpool status                                              love: Fri Apr 30 07:52:40 2021

  pool: data
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: scrub in progress since Thu Apr 29 02:30:54 2021
        4.32T scanned out of 5.03T at 42.9M/s, 4h48m to go
        466K repaired, 85.93% done
config:

        NAME         STATE     READ WRITE CKSUM
        data         ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            sda      ONLINE       0     0     0
            sdb      ONLINE       0     0     0
          mirror-1   ONLINE       0     0     0
            sdc      ONLINE       0     0     4  (repairing)
            sdd      ONLINE       0     0     0
        cache
          nvme0n1p4  ONLINE       0     0     0

errors: No known data errors

Modify reserved disk space quota

I didn’t know you could do this so easily. Linux file systems reserve space for the super user. But that can just be a waste of 5% of your useful space, particularly on removable drives. Anyway I was reading about fdisk and mkfs over on InstallingANewHardDrive and discovered that if you want to reserve no space (i.e. have all your space available for a user) then you can just run the following command:

  tune2fs -m 0 /dev/sdb1

Where /dev/sdb1 is the partition you are modifying. Handy!