[fixed/closed] kernel (lts) 6.1.0-15 - kernels 6.2.++ zfs problems ...  [Solved]

Report Bugs, Issues and non- package Requests
Forum rules
Post Reply
Message
Author
User avatar
l0dr3
Posts: 463
Joined: Wed Jun 28, 2023 11:06 am

[fixed/closed] kernel (lts) 6.1.0-15 - kernels 6.2.++ zfs problems ...

#1 Post by l0dr3 »

I just saw this official warning by @dolphin_oracle D_O ...

viewtopic.php?p=756154#p756154

UPDATE: 2023/12/14 - 07:45

I had serious trouble over WE with this updated LTS 6.1.0-15 (6.1.66) LTS 6.1.0-14 (6.1.65) kernel on two production setups, tried to upgrade the setups to 6.5 LTS-Backports by Debian kernels (and also tried w#6.6.4 custom compiled ...), w# setups, that had ZFS-DKMS enabled AND: worked well on LTS kernels 6.1.0-13 before!

TL;DR:

When you are using 'ZFS-on-Linux' in production, avoid to update to any >=6.6.++ kernel variant,
when 6.1.66 6.1.64 or 6.1.65 current EXT4 BUGS stresses you, stay on 6.1.0-13, if applicable :crossfingers:

Solution found for the bug-fixed ZFS-DKMS 2.1.14 and official debian backport 6.5.10 STABLE kernel
viewtopic.php?p=756393#p756393
and 6.5.13 TRIXIE/TESTING kernel
viewtopic.php?p=756460#p756460

:blindfold:
Here are some notes by me, archived over the weekend, regarding BUGs, triggered by 'coreutils v9.x.y cp', in ZoL <=2.1.14-bpo12u2, and the fix in 2.2.2-experimental only thus far (so not really a solution for InProd setups :rolleyes: )
sudo apt search linux-headers |grep installed
linux-headers-6.5.0-0.deb12.4-amd64/stable-backports,now 6.5.10-1~bpo12+1 amd64 [installed,automatic]
linux-headers-6.5.0-0.deb12.4-common/stable-backports,stable-backports,now 6.5.10-1~bpo12+1 all [installed,automatic]
linux-headers-6.6.2-humpie-1/now 6.6.2-humpie-1-4 amd64 [installed,local]
linux-headers-6.6.4-humpie-2/now 6.6.4-humpie-2-1 amd64 [installed,local]
linux-headers-amd64/stable-backports,now 6.5.10-1~bpo12+1 amd64 [installed]
OK!

sudo apt install zfs-dkms zfs-zed zfsutils-linux

Oooops!!
Building initial module for 6.6.4-humpie-2
configure: error:
*** None of the expected "bops->release()" interfaces were detected.
*** This may be because your kernel version is newer than what is
*** supported, or you are using a patched custom kernel with
*** incompatible modifications.
***
*** ZFS Version: zfs-2.1.11-1
*** Compatible Kernels: 3.10 - 6.2

On MX Linux, CleanUp unused / unnecessary kernels ..., then

sudo apt purge zfs-dkms zfs-zed zfsutils-linux && sudo apt autoremove
sudo apt install -t bookworm-backports zfs-dkms zfs-zed zfsutils-linux

Oooops2: still .. errors for kernel 6.6.4-humpie-2 !?
Loading new zfs-2.1.14 DKMS files...
Building for 6.6.4-humpie-2
Building initial module for 6.6.4-humpie-2
Error! Bad return status for module build on kernel: 6.6.4-humpie-2 (x86_64)
Consult /var/lib/dkms/zfs/2.1.14/build/make.log for more information.
Error! Bad return status for module build on kernel: 6.6.4-humpie-2 (x86_64)
Consult /var/lib/dkms/zfs/2.1.14/build/make.log for more information.
Done.
Setting up libuutil3linux (2.1.14-1~bpo12+1) ...
Setting up libzfs4linux (2.1.14-1~bpo12+1) ...
Setting up libzpool5linux (2.1.14-1~bpo12+1) ...
Setting up zfsutils-linux (2.1.14-1~bpo12+1) ...
modprobe: FATAL: Module zfs not found in directory /lib/modules/6.6.4-humpie-2
modprobe: FATAL: Module zfs not found in directory /lib/modules/6.6.4-humpie-2
invoke-rc.d: initscript zfs-import, action "start" failed.
dpkg: error processing package zfsutils-linux (--configure):
installed zfsutils-linux package post-installation script subprocess returned error exit status 1
Processing triggers for libc-bin (2.36-9+deb12u3) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for initramfs-tools (0.142) ...
update-initramfs: Generating /boot/initrd.img-6.6.4-humpie-2
Errors were encountered while processing:
zfsutils-linux

Aaaargh :( - g00gled - found
https://bugs.debian.org/cgi-bin/bugrepo ... ug=1057452
:
Package: zfs-dkms
Version: 2.1.14-1
Severity: important
Control: close -1 2.2.2-1~exp1

Hi,

zfs-dkms fails to build a module for Linux 6.6 that was just uploaded
to experimental
:
Date: Tue, 5 Dec 2023 10:21:01 UTC
Severity: important
Found in version zfs-linux/2.1.14-1
Fixed in version zfs-linux/2.2.2-1~exp1
sudo apt purge zfs-dkms zfsutils-linux && sudo apt autoremove

situation asof 2023/12/12 - 11:00

https://forums.debian.net/viewtopic.php?t=157693

for further reference, see:

https://www.phoronix.com/forums/forum/s ... ssue/page5
https://forums.debian.net/viewtopic.php?t=157693
https://bugs.debian.org/cgi-bin/bugrepo ... =1057452#8
:
et al++ (g00gle!)

BE CAREFUL, stay well ... greetz, l0dr3
Last edited by l0dr3 on Thu Dec 14, 2023 2:24 am, edited 3 times in total.
Top
User avatar
l0dr3
Posts: 463
Joined: Wed Jun 28, 2023 11:06 am

Re: kernel (lts) 6.1.0-15 - kernels 6.2.++ zfs problems ...

#2 Post by l0dr3 »

Update for the ZFS situation asof 2023/12/12 - 14:45:

Code: Select all

$ uname -a
Linux mx 6.1.0-15-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.66-1 (2023-12-09) x86_64 GNU/Linux

$ apt policy zfsutils-linux
zfsutils-linux:
  Installed: 2.1.14-1~bpo12+1
  Candidate: 2.1.14-1~bpo12+1
  Version table:
 *** 2.1.14-1~bpo12+1 100
        100 /var/lib/dpkg/status
     2.1.11-1 500
        500 http://deb.debian.org/debian bookworm/contrib amd64 Packages
for reference, what was wrong with the released kernels over the last WE ...
https://bugs.debian.org/cgi-bin/bugrepo ... 1057843#78
++ for reference, what is the status-quo of debian point-releases asof 2023/12/12
https://www.debian.org/News/2023/20231210

So - for the 6.1.0-LTS kernel series, problem solved :crossfingers: (for my two production setups)
ZFS-DKMS compiles w#o errors.

On my TESTING-VM (SID w#kernel 6.6.6 ;) ), updated this morning, ZFS-DKMS compile still fails for the modules-release 2.1.14-1

On my TESTING-BOX, MX23.1 custom kernel compile for 6.6.6 is underway ...

UPDATE: 2023/12/14 - 07:45

will temporary enable EXPERIMENTAL on that box to get modules-release 2.2.2,
will report the results later that day ...

wait for modules-release 2.2.2 in SID, TRIXIE and BOOKWORM-BACKPORTS :needcoffee:

Solutions found for BOOKWORM-BACKPORTS and TRIXIE/TESTING kernels, see posts below ...

greetz l0dr3
Last edited by l0dr3 on Thu Dec 14, 2023 2:05 am, edited 1 time in total.
Top
User avatar
Stevo
Developer
Posts: 14843
Joined: Fri Dec 15, 2006 7:07 pm

Re: kernel (lts) 6.1.0-15 - kernels 6.2.++ zfs problems ...

#3 Post by Stevo »

Code: Select all

/lib/modules/6.6.4-humpie-2
Inquiring minds would appreciate knowing where this kernel came from, and if it was compiled with the same gcc as the installed Linux.

I.e. xanmod kernels compiled with gcc-13 will boot and run on MX 23, but will bail out when trying to build any DKMS packages at all.
MXPI = MX Package Installer
QSI = Quick System Info from menu
The MX Test repository is mostly backports; not the same as Debian testing
Top
User avatar
l0dr3
Posts: 463
Joined: Wed Jun 28, 2023 11:06 am

Re: kernel (lts) 6.1.0-15 - kernels 6.2.++ zfs problems ...

#4 Post by l0dr3 »

Stevo wrote: Wed Dec 13, 2023 2:21 am

Code: Select all

/lib/modules/6.6.4-humpie-2
Inquiring minds would appreciate knowing where this kernel came from, and if it was compiled with the same gcc as the installed Linux.

I.e. xanmod kernels compiled with gcc-13 will boot and run on MX 23, but will bail out when trying to build any DKMS packages at all.
This is just a renamed 6.6.4 stock kernel from last week, i compiled for my own development needs, with some special flags enabled :alien:

It was a little bit late yesterday evening to post my further findings on this zfs issues, heres the status-quo asof 2023/12/13 - 14:30

TL;DR

Debian has bookworm-backported versions for the zfsutils-linux, zfs-dkms and other zfs-* stuff in version '2.1.14-1~bpo12+1: all', that addresses the data-corruption bug, when using 'cp' of coreutils 9.x.y (see phoronix https://www.phoronix.com/news/OpenZFS-2.2.2-Released). In fact, over the last week/weekend they had 2 showstopper data-corruption bugs, a ext4-bug and a zfs-bug at same time :mad:

The ZFS-DKMS install/builds failed for me not only for any 6.6-series kernel but also with their latest backport kernel Debian 6.5.10-1~bpo12+1 (2023-11-23) :confused:
After purging out all of the EXT4-buggy 6.1.xx (64 and 65 ones) the LTS 6.1.66 from WE was the only kernel that supported ZFS for me, but: i need at least a 6.4++ for the 3 BE EQ12 systems, i'm preparing for ZFS driven Home-NAS-Server usage (bcof their Q1/23 Intel AX101 WIFI chipsets!)

BUT this morning, i sorted it out: I now have a working setup for ZFS w#official Debian 6.5.10-1~bpo12+1 (2023-11-23) for my TESTING-BOXes w#full support for the AX101 WiFi chipsets :happy:

I'll post the whole procedures (a huge .txt-file, i wont put in code-tags here) for the interested and curious as attachment below later that day.
EDIT: 2023/12/14 - 07:45
Here it is - this notes were taken with cherrytree, so for preserving the bolded annotations and some screenshots, heres a .pdf.
2023--December--13_Wed--MX23#_6.6.6++zfs_kernel_compilations.pdf
greetz l0dr3
You do not have the required permissions to view the files attached to this post.
Last edited by l0dr3 on Thu Dec 14, 2023 1:47 am, edited 1 time in total.
Top
User avatar
l0dr3
Posts: 463
Joined: Wed Jun 28, 2023 11:06 am

Re: kernel (lts) 6.1.0-15 - kernels 6.2.++ zfs problems ...

#5 Post by l0dr3 »

Status-Report asof : 2023/12/13 - 21:00

For the latest, but EOLed, kernel from series 6.5.xx ... 6.5.13 current TRIXIE/TESTING release ...

Code: Select all

mkdir -p ~/kernels/stable/6.5.13
cd ~/kernels/stable/6.5.13
wget 'https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.13.tar.xz'
tar -xaf linux-6.5.13.tar.xz
cd linux-6.5.13
cp /boot/config-6.5.0-0.deb12.4-amd64 ./.config
:
2023/12/13 - 16:30
make bindeb-pkg
2023/12/13 - 20:00
:
sudo apt-get install ./linux-headers-6.5.13_6.5.13-1_amd64.deb \
./linux-image-6.5.13_6.5.13-1_amd64.deb
:
Setting up linux-headers-6.5.13 (6.5.13-1) ...
Setting up linux-image-6.5.13 (6.5.13-1) ...
dkms: running auto installation service for kernel 6.5.13.
/usr/sbin/dkms.mx autoinstall --kernelver 6.5.13
Sign command: /lib/modules/6.5.13/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub
:
Building module:
Cleaning build area...
make -j4 KERNELRELEASE=6.5.13..............................
Signing module /var/lib/dkms/zfs/2.1.14/build/module/avl/zavl.ko
Signing module /var/lib/dkms/zfs/2.1.14/build/module/nvpair/znvpair.ko
Signing module /var/lib/dkms/zfs/2.1.14/build/module/unicode/zunicode.ko
Signing module /var/lib/dkms/zfs/2.1.14/build/module/zcommon/zcommon.ko
Signing module /var/lib/dkms/zfs/2.1.14/build/module/zfs/zfs.ko
Signing module /var/lib/dkms/zfs/2.1.14/build/module/icp/icp.ko
Signing module /var/lib/dkms/zfs/2.1.14/build/module/lua/zlua.ko
Signing module /var/lib/dkms/zfs/2.1.14/build/module/spl/spl.ko
Signing module /var/lib/dkms/zfs/2.1.14/build/module/zstd/zzstd.ko

Running the post_build script:
Cleaning build area...

:
zcommon.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.5.13/updates/dkms/

zfs.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/6.5.13/updates/dkms/
:
depmod...
dkms: autoinstall for kernel: 6.5.13.
update-initramfs: Generating /boot/initrd.img-6.5.13
Generating grub configuration file ...
Found theme: /boot/grub/themes/mx_linux/theme.txt
Found linux image: /boot/vmlinuz-6.6.6-humpie-3
Found initrd image: /boot/initrd.img-6.6.6-humpie-3
Found linux image: /boot/vmlinuz-6.6.4-humpie-2
Found initrd image: /boot/initrd.img-6.6.4-humpie-2
Found linux image: /boot/vmlinuz-6.5.13
Found initrd image: /boot/initrd.img-6.5.13
Found linux image: /boot/vmlinuz-6.5.0-0.deb12.4-amd64
Found initrd image: /boot/initrd.img-6.5.0-0.deb12.4-amd64
Found mtest-64.efi image: /boot/uefi-mt/mtest-64.efi
Found Windows Boot Manager on /dev/nvme0n1p1@/EFI/Microsoft/Boot/bootmgfw.efi
Adding boot menu entry for EFI firmware configuration
done
:
!! Looks good ... :crossfingers:

greetz l0dr3
Top
User avatar
l0dr3
Posts: 463
Joined: Wed Jun 28, 2023 11:06 am

[fixed/closed] kernel (lts) 6.1.0-15 - kernels 6.2.++ zfs problems ...  [Solved]

#6 Post by l0dr3 »

2023/12/14 - 08:00 UPDATES applied to entry-post ff.. see my edits above.

Conclusion:

According to Debian latest infos https://www.debian.org/News/2023/20231210

EXT4 BUG [fixed/closed] in LTS kernel releases >=6.1.66 (6.1.0-15++)

ZFS-DKMS bug-fix-release '2.1.14-1~bpo12+1: all' for 'coreutils' packages v9.x.y applicable on all current OFFICIAL SUPPORTED debian releases
BOOKWORM(-BACKPORTS), TRIXIE/TESTING
bullseye(oldstable) and buster(oldoldstable) are not affected by this specific zfs-bug bcof 'coreutils' packages v8.x.y see: https://packages.debian.org/search?suit ... =coreutils
SID not tested yet :needcoffee: :needcoffee:

Marking this thread now as [fixed/closed] and [solved] for ME :happy:

greetz l0dr3
Top
Post Reply

Return to “Bugs and Non-Package Requests Forum”