I had a long running ZFS snapshot that was sending data from the cloud to my local ZFS box, and when it was just about completed, I received a message

cannot receive resume stream: kernel modules must be upgraded to receive this stream.

This was somewhat cryptic, as I didn’t think I had any kernel modules that would be affecting this. A small bit of research led me to this GitHub issue:

https://github.com/openzfs/zfs/issues/7024

Basically, the issue was that due to the long running nature of this, my auto-snapshot creating rules had (un)helpfully created some extra snapshots for me on the receiving side.

A quick zfs list -t snapshot Master/ocean confirmed this for me:

root@Iridium[~] zfs list -t snapshot Master/ocean
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
Master/ocean@now                       0B      -      176M  -
Master/ocean@auto-2023-02-22_18-00     0B      -      176M  -
Master/ocean@auto-2023-02-23_00-00     0B      -      176M  -
Master/ocean@auto-2023-02-23_06-00     0B      -      176M  -

I proceeded to destroy these (as I didn’t need them in my case), and resumed the transfer:

root@Iridium[~] zfs destroy Master/ocean@auto-2023-02-23_18-00
# repeat for each snapshot
root@Iridium[~] ssh [email protected] zfs send -wvP -t <RESUME_TOKEN> | zfs recv -s -v Master/ocean
resume token contents:
nvlist version: 0
        fromguid = 0xCENSORED
        object = 0xCENSORED
        offset = 0xCENSORED
        bytes = 0xCENSORED
        toguid = 0xCENSORED
        toname = ocean/local@now-2
        compressok = 1
        rawok = 1
incremental     ocean/local@now ocean/local@now-2       0
receiving incremental stream of ocean/local@now-2 into Master/ocean@now-2

I was presented with another annoying error though:

cannot receive resume stream: destination Master/ocean has been modified since most recent snapshot

So I then had to roll back to the original snapshot, before completing the recv

root@Iridium[~] zfs rollback Master/ocean@now
root@Iridium[~] ssh [email protected] zfs send -wvP -t <RESUME_TOKEN> | zfs recv -s -v Master/ocean
resume token contents:
nvlist version: 0
        fromguid = 0xCENSORED
        object = 0xCENSORED
        offset = 0xCENSORED
        bytes = 0xCENSORED
        toguid = 0xCENSORED
        toname = ocean/local@now-2
        compressok = 1
        rawok = 1
incremental     ocean/local@now ocean/local@now-2       0
receiving incremental stream of ocean/local@now-2 into Master/ocean@now-2
received 132K stream in 1 seconds (132K/sec)
root@Iridium[~] zfs list -t snapshot Master/ocean
NAME                 USED  AVAIL     REFER  MOUNTPOINT
Master/ocean@now     202K      -      176M  -
Master/ocean@now-2     0B      -     1.63T  -

##Confirming

root@Iridium[~] zfs list -t snapshot Master/ocean
NAME                 USED  AVAIL     REFER  MOUNTPOINT
Master/ocean@now     202K      -      176M  -
Master/ocean@now-2     0B      -     1.63T  -
root@Iridium[~] zfs load-key Master/ocean
Enter passphrase for 'Master/ocean':
root@Iridium[~] zfs mount Master/ocean
root@Iridium[~] ls /mnt/Master/ocean
bear-notes-backups      minio                   standard-notes-backups

Figured this may help someone else in the same situation 🙂