MX Linux Forum

Important Alert
Do not upgrade your MX 17 or MX18 system installed or live until further notice. A problem with a kernel upgrade has occurred. A fix will be posted when available.

I unfortunately upgraded today, as there is no alert messages in MXPackae Manager so I didnt see the alert until reading it on the usergroup.
But didnt reboot yet.

What is the nature of the problem and how can I undo the kernel upgrade without breaking things.
Luckily I didnt reboot.

Would be great to get some help to get the previous kernel reinstalled if the new kernel is going to break the system.

Thanks

See
viewtopic.php?f=65&t=51104
for more info...

In Thunar go to File System, then /usr/share/mx-packageinstaller-pkglist/, then open a terminal there and run and it'll be all fixed.

Im not allowed to attach files it seems so I post the output.
I cannot see that the new kernel has been removed and the old one used in its place.

When is it anticipated that a fix will be available during an upgrade from MX-Package-Manager ?
It might be best not to boot till then.




# sudo ./rebuild_dkms_packages.sh

-------- Uninstall Beginning --------
Module: broadcom-sta
Version: 6.30.223.271
Kernel: 4.19.0-1-amd64 (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

wl.ko:
- Uninstallation
- Deleting from: /lib/modules/4.19.0-1-amd64/updates/dkms/
- Original module
- No original module was found for this module on this kernel.
- Use the dkms install command to reinstall any previous module version.

depmod...

Backing up initrd.img-4.19.0-1-amd64 to /boot/initrd.img-4.19.0-1-amd64.old-dkms
Making new initrd.img-4.19.0-1-amd64
(If next boot fails, revert to initrd.img-4.19.0-1-amd64.old-dkms image)
update-initramfs........

DKMS: uninstall completed.

-------- Uninstall Beginning --------
Module: broadcom-sta
Version: 6.30.223.271
Kernel: 4.19.0-5-amd64 (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

wl.ko:
- Uninstallation
- Deleting from: /lib/modules/4.19.0-5-amd64/updates/dkms/
- Original module
- No original module was found for this module on this kernel.
- Use the dkms install command to reinstall any previous module version.

depmod.....

Backing up initrd.img-4.19.0-5-amd64 to /boot/initrd.img-4.19.0-5-amd64.old-dkms
Making new initrd.img-4.19.0-5-amd64
(If next boot fails, revert to initrd.img-4.19.0-5-amd64.old-dkms image)
update-initramfs........

DKMS: uninstall completed.

------------------------------
Deleting module version: 6.30.223.271
completely from the DKMS tree.
------------------------------
Done.
Loading new broadcom-sta-6.30.223.271 DKMS files...
Building for 4.19.0-1-amd64 4.19.0-5-amd64
Building initial module for 4.19.0-1-amd64
Done.

wl:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/4.19.0-1-amd64/updates/dkms/

depmod...

Backing up initrd.img-4.19.0-1-amd64 to /boot/initrd.img-4.19.0-1-amd64.old-dkms
Making new initrd.img-4.19.0-1-amd64
(If next boot fails, revert to initrd.img-4.19.0-1-amd64.old-dkms image)
update-initramfs........

DKMS: install completed.
Building initial module for 4.19.0-5-amd64
Done.

wl:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/4.19.0-5-amd64/updates/dkms/

depmod...

Backing up initrd.img-4.19.0-5-amd64 to /boot/initrd.img-4.19.0-5-amd64.old-dkms
Making new initrd.img-4.19.0-5-amd64
(If next boot fails, revert to initrd.img-4.19.0-5-amd64.old-dkms image)
update-initramfs........

DKMS: install completed.

-------- Uninstall Beginning --------
Module: ndiswrapper
Version: 1.61
Kernel: 4.19.0-1-amd64 (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

ndiswrapper.ko:
- Uninstallation
- Deleting from: /lib/modules/4.19.0-1-amd64/updates/
- Original module
- No original module was found for this module on this kernel.
- Use the dkms install command to reinstall any previous module version.
/usr/sbin/dkms: line 2007: echo: write error: Broken pipe
/usr/sbin/dkms: line 2009: echo: write error: Broken pipe

depmod...

DKMS: uninstall completed.

-------- Uninstall Beginning --------
Module: ndiswrapper
Version: 1.61
Kernel: 4.19.0-5-amd64 (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

ndiswrapper.ko:
- Uninstallation
- Deleting from: /lib/modules/4.19.0-5-amd64/updates/
- Original module
- No original module was found for this module on this kernel.
- Use the dkms install command to reinstall any previous module version.

depmod...

DKMS: uninstall completed.

------------------------------
Deleting module version: 1.61
completely from the DKMS tree.
------------------------------
Done.
Loading new ndiswrapper-1.61 DKMS files...
Building for 4.19.0-1-amd64 4.19.0-5-amd64
Building initial module for 4.19.0-1-amd64
Done.

ndiswrapper:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/4.19.0-1-amd64/updates/

depmod...

DKMS: install completed.
Building initial module for 4.19.0-5-amd64
Done.

ndiswrapper:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/4.19.0-5-amd64/updates/

depmod...

DKMS: install completed.

-------- Uninstall Beginning --------
Module: vhba
Version: 20170610
Kernel: 4.19.0-1-amd64 (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

vhba.ko:
- Uninstallation
- Deleting from: /lib/modules/4.19.0-1-amd64/
rmdir: failed to remove '': No such file or directory
- Original module
- No original module was found for this module on this kernel.
- Use the dkms install command to reinstall any previous module version.
/usr/sbin/dkms: line 2007: echo: write error: Broken pipe
/usr/sbin/dkms: line 2009: echo: write error: Broken pipe

depmod...

DKMS: uninstall completed.

-------- Uninstall Beginning --------
Module: vhba
Version: 20170610
Kernel: 4.19.0-5-amd64 (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

vhba.ko:
- Uninstallation
- Deleting from: /lib/modules/4.19.0-5-amd64/kernel/updates/dkms/
- Original module
- No original module was found for this module on this kernel.
- Use the dkms install command to reinstall any previous module version.

depmod...

DKMS: uninstall completed.

------------------------------
Deleting module version: 20170610
completely from the DKMS tree.
------------------------------
Done.
modprobe: FATAL: Module vhba is in use.
WARNING: Failed to unload running module.
Loading new vhba-20170610 DKMS files...
Building for 4.19.0-1-amd64 4.19.0-5-amd64
Building initial module for 4.19.0-1-amd64
Done.

vhba:
Running module version sanity check.

Good news! Module version 20170610 for vhba.ko
exactly matches what is already found in kernel 4.19.0-1-amd64.
DKMS will not replace this module.
You may override by specifying --force.

depmod...

DKMS: install completed.
Building initial module for 4.19.0-5-amd64
Done.

vhba:
Running module version sanity check.
- Original module
- No original module exists within this kernel
- Installation
- Installing to /lib/modules/4.19.0-5-amd64/kernel/updates/dkms/

depmod...

DKMS: install completed.
modprobe: FATAL: Module vhba is in use.
WARNING: Failed to unload running module.
WARNING: Configuration already exists.

KISS - Love it - Bravo

Thank you & regards
Jack

JayM wrote: Sat Jun 22, 2019 6:16 am In Thunar go to File System, then /usr/share/mx-packageinstaller-pkglist/, then open a terminal there and run

Code: Select all

sudo ./rebuild_dkms_packages.sh

and it'll be all fixed.

@zimbodel,

Please place the text output within the < / > brackets that you can see in the editing toolbar in the forum.

If you already installed the updates to the 4.19 kernel, you won't see that the kernel has been removed in Synaptic. The removed package was an update to the same 4.19 kernel with the same version number (that was what caused the issue).

So since you have already installed the updates, you can only rebuild modules for this existing kernel in your system and/or install another kernel (DEbian Stable 4.9, MXPI Popular apps) or the latest 5.1+ liquorix (MXPI Test Repo).

If a user hadn't yet installed all the updates before the hooha, when they now check for updates, they won't find any update to that 4.19 kernel in the list of updates, which means they are still on their old, no-issue 4.19 kernel.

But I'm afraid I don't know enough to read your output. YOu'll have to wait for the experts to wake up in their timezone.

Have you even rebooted to see if it works? If not, I suggest you take the opportunity to install another kernel first, as suggested above. Then reboot and see what happens. If it doesn't work, you'll have another kernel(s) to boot into.

BTW, I sometimes get alarming warnings when I install new liquorix kernels, but after rebooting, it's fine.

I dont want to boot as what I am doing on this system is too crytical to risk having downtime.

From what you say, it seems that all the advice posted is not going to help...

Ok, so where do I find an older(previous) or the correct kernel. ?

I have no idea where to find it.

I guess I need a link and instructi9ons to the kernel just before this broken one.

This is what is running, and should be the old kernel that worked spectacularly well.

Linux version 4.19.0-1-amd64 (stevep@mxlinux.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP Debian 4.19.5-2~mx17+1 (2018-12-12)

So how do I get back to this.

zimbodel wrote: Fri Jun 28, 2019 10:12 pm I dont want to boot as what I am doing on this system is too crytical to risk having downtime.

From what you say, it seems that all the advice posted is not going to help...

Ok, so where do I find an older(previous) or the correct kernel. ?

I have no idea where to find it.

I guess I need a link and instructi9ons to the kernel just before this broken one.

This is what is running, and should be the old kernel that worked spectacularly well.

Linux version 4.19.0-1-amd64 (stevep@mxlinux.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP Debian 4.19.5-2~mx17+1 (2018-12-12)

So how do I get back to this.

The problem with the 4.19.05 kernel was resolved over a week ago and the red alert message removed from the forum. There's an announcement in the MX News & Announcements forum.

If you still want to change kernels just install the one you want, then in MX Boot Options check the checkbox by "Enable saving last boot choice" then click Apply then Close, reboot, and select the kernel you want from the grub boot menu.

If you want to prevent a kernel from being updated (not recommended as you'll miss out on critical security and bugfix patches) you can easily do that in Synaptic Package Manager. Search for your kernel, which should be named starting with "linux-image", select it, then click the Package dropdown menu and click Lock Version.

The issue with 4.19.05 was a one time "oopsie" by the upstream (Debian Stable.) This is the first time in almost 20 years of using Linux that I've seen this happen with a stable kernel from a reliable source. I wouldn't worry about it happening again as the odds are slim to none. So feel free to install the recommended updates and/or reboot whenever you want. It's safe now.

I dont blame anyone.

Is there any way to upgrade to the new kernel that has been corrected ? given I still have the defect one installed ?
If it is as easy as going to Mx-package manager and just install the kernel update it might be the easiest way.

I went to look in the package manager and I could only find linu8x=-source 4.19+105 as upgradeable which is listed as linux kernel source. this was in the MX Test Repo.

Do you maybe have the exact kernel version I should upgrade to unless debian foolishly named it exact the same which would be a disaster. then I will have to downgrade to my previous version as listed above in a previous post.

What I have in MX-Packagemanager which is upgradeable is:
Stable Repo: Linux-source-4.9 Version 4.9.168-1+deb9u3
Mx Test Repo; 4.19+105
Debian Backports: 4.19+105~bp09+1

Why is the kernel in the test repo a lower issue number that the stable !? Doesnt seem right !


Note if I run apt list upgradeable I get
linux-headers-4.9.0-9-amd64/stable 4.9.168-1+deb9u3 amd64 [upgradable from: 4.9.168-1+deb9u2]
linux-headers-4.9.0-9-common/stable,stable 4.9.168-1+deb9u3 all [upgradable from: 4.9.168-1+deb9u2]
linux-kbuild-4.9/stable 4.9.168-1+deb9u3 amd64 [upgradable from: 4.9.168-1+deb9u2]
linux-source-4.9/stable,stable 4.9.168-1+deb9u3 all [upgradable from: 4.9.168-1+deb9u2]

Would upgrading to stable 4.9.168-1+deb9u3 solve the problem ??

What should my strategy be for the three options, back or forward ?
Kernel problems are serious an difficult to fix once it freezes, so I rather ask dead dumb questions to make sure I get the correct kernel installed that will at least boot.

I have no idea what problem you're trying to solve by changing to an older kernel, since you didn't mention that something on your computer wasn't working.

The MX Update Manager will upgrade your 4.19.0-1 kernel to 4.19.0-5 for you as long as you haven't locked or pinned the 0-1 kernel in Synaptic as I explained how earlier. If you did, just un-pin it.

I think zimbodel just wants a secondary kernel installed so that when he reboots - which he hasn't done since the 4.19 kernel issue - he has something that works if his 4.19 doesn't.

If I'm correct in my understanding, zimbodel, the first step is to paste the output of MX Tools >> Quick system info here so that we know your hardware specs.

If it's seen to be older hardware, then installing Debian Stretch 's default 4.9 kernel from MXPI>>Popular Apps>> kernel will be a good bet. If it's much newer hardware, I would recommend the latest Liquorix 5+ kernel from MXPI>> Test Repo tab.

One last question: right now do you only have 1 kernel installed or have you already installed others?

Check in Thunar: /boot folder

how many different vmlinuz files do you have?

asqwerth wrote: Sat Jun 29, 2019 4:27 am I think zimbodel just wants a secondary kernel installed so that when he reboots - which he hasn't done since the 4.19 kernel issue - he has something that works if his 4.19 doesn't.

Ah, OK, that makes sense. Just having a backup kernel available, just in case. That's actually a very good idea.

Maybe MX-19 (or 18.4 if there will be one) could ship with the 4.9 kernel also preinstalled but not configured to boot by default? There are a few people with older hardware that are already having problems with 4.19 (Yours Truly included) so it would be nice if an alternative, older kernel was already available to them without having to be installed first (which they can't do if they can't boot or have frequent lock-ups), so if they had kernel issues they could fall back to 4.9. It would also be there in case something ever goes wrong with the default kernel again. That is, if a second kernel wouldn't bloat the .iso too much.

The live kernel updater plus the antiX CLI apt tools (preinstalled in MX) do allow you to install a new kernel on your live-running USB when it's in non graphical mode. Never actually tried it myself. T'was the basis for fehlix's fix for some members whose live USB persistent install got hit by the kernel issue, IIUC.

Only thing is that most people probably won't know about it unless they read up or ask in the forum.

Or have read the Blog post (MX Home > News) that was linked to social media and reposted to over 10,000 users.

If I were to suggest to our users how to keep up on the latest developments, it would be to subscribe to the Blog and follow us on Twitter. Wonder if we could add that to the last screen of the Installer...?

asqwerth wrote: Sat Jun 29, 2019 5:40 am The live kernel updater plus the antiX CLI apt tools (preinstalled in MX) do allow you to install a new kernel on your live-running USB when it's in non graphical mode. Never actually tried it myself.

I tried it myself a day of two ago but wasn't successful for various reasons, mostly to do with the USB stick I was using, I think. I'll try it again when I get time.

My thought was actually to include both the 4.19.x and 4.9.x kernels in the MX iso, with 4.19 booting by default but 4.9 also being available to boot from in grub's advanced options, so if a newbie needed the older kernel it would be there already. It would be much easier to tell him how to boot with 4.9 instead of 4.19 vs. talking him through installing 4.9 on a persistent USB, remastering, then running the live kernel updater. That's a lot of technical hoops to ask a Linux newbie to jump through, as opposed to the 4.9 kernel being already preinstalled and able to be booted when needed. I think there are enough users with older hardware having lock-up and other issues with the 4.19 kernel, judging by the number of posts I've seen where the older kernel is being recommended for old systems, to warrant also making 4.9 available on the Live USB. Again, if having two kernels preinstalled wouldn't bloat the ISO or Live USB too much.

It could also be a "selling point" as it were. The MX web site, under Products/Current Release Features, says

Kernels (secured against known vulnerabilities)

32bit: Linux kernel 4.19.5 PAE (non-PAE kernel: antiX). Very stable kernel for many machines.
64bit: linux-image-4.19.5-amd64. Recent stable kernel for newer machines.
Easy kernel upgrade or downgrade with MX Package Installer

It could also add that Debian Linux kernel 4.9 is also preinstalled and available for older machines that may have issues with newer kernels.

Swings!

It is a good idea, I agree.

But as much as I can understand it, the live system has a different bootloader I think. For the installed system, Grub has the menu from which you can select between kernels. I don't know if you can actually do that in live system.

Perhaps go to MX Bugzilla and propose this as a new feature?

I already know what Jerry's going to say when he reads this so I've already submitted it as an enhancement request in the MX/antiX Bugzilla.

Great idea, and terrific use of Bugzilla.

asqwerth wrote: Sat Jun 29, 2019 8:23 am It is a good idea, I agree.

But as much as I can understand it, the live system has a different bootloader I think. For the installed system, Grub has the menu from which you can select between kernels. I don't know if you can actually do that in live system.

Perhaps go to MX Bugzilla and propose this as a new feature?

Oh, you're right. I forgot that the live USB doesn't use grub. I added a comment in bugzilla to that effect. It will entail allowing a choice between kernels in the USB's bootloader which adds complexity to my requested enhancement.
(Sorry.)

Now I am confused.
Why is it so difficult to understand that
1) I upgraded to the defect kernel
2) I dont want to reboot as a non booting system is a horrror to fix, and I am not convinced I wll have another kernel available through grub at boot. I just CANNOT know.
3) I use/need this system 24/7 so I dont want to risk booting it and rather want to make sure I reinstall a kernel that WILL at least boot.
4) I am not sure if the latest kernel update that appeared after the warning on the site was removed, in the MX-Package manager will in fact remove the defunct kernel and install a new corrected working kernel.
So which is it. Only the developers can know that even if I have an IQ of 205.

It is easy............

zimbodel wrote: Sat Jun 29, 2019 10:40 pm ... and I am not convinced I wll have another kernel available through grub at boot. I just CANNOT know.

Zimbodel, read this
viewtopic.php?p=512349#p512349

And respond.

I was trying to ascertain what kernels you have and to provide you with a backup kernel if you don't have others.

4) I am not sure if the latest kernel update that appeared after the warning on the site was removed, in the MX-Package manager will in fact remove the defunct kernel and install a new corrected working kernel.

I don't believe it does. It was not an update but a removal of the bad update but this only helps those who haven't installed the bad update yet. For those who installed the bad, some won't be affected by it if they're not using modules that have to be rebuilt. For those who were affected, the fix that was posted was to trigger the rebuilding of the modules.

We don't know which group you are because you've never rebooted. I believe you tried the fix but are still afraid of putting it to the test.

So pls respond to my post so we can suggest the appropriate backup kernel for you and you can proceed to reboot with the main kernel, and if it doesn't, the backup.

For the rest of the posters, pls don't go off-topic again and confuse him.

Thank you for trying to help.

I will update this post for the next few hours as I read through what you require.

-------------------------------------------------------------------------
To the question asked in the link:
Also,
1) yes I did install the defect kernel during an upgrade and I DID NOT reboot since I saw the alert luckily before doing so.
2) I am not sure if the previous kernel will be available in the boot grub interface. If not Im likely screwed.
3) What I try to do is to install the previous kernel again, which I already posted and which is still running as I did not boot yet.
I dont know why it is so impossible to install the previous kernel I replaced during the kernel upgrade ?
4) I also did as suggested and it executed without any error.
I am not sure what it fixed at all and whether it recovered my previous kernel. What did it actually do ?

5) For reference: Again, the kernel I want to roll back to is. Which is currently still running as I did not reboot.

6) Again, I cannot just simply reboot and check. This server is critical, and I am caught bad by the defect kernel. If I have to run this server for 3 years without booting like this it is fine, until I can get a new kernel update without defect later. Is that an option ?

Most of my servers are running 6-8 months without any need for reboot. It is usually powerblackouts and lightning blackouts that exceeds UPS life that forces a reboot.


I hope this makes it clear.

Zimbodel, is there a reason you will not even install a fallback kernel? Or post the Quick System Info of your system hardware?

[added] Since you said you have run the fix, I really don't know how to tell you whether it works or not if you don't try a reboot. YOu asked what the fix did, but I already explained in my last post that it triggers the rebuilding of modules, which the bad kernel update did not do. That should fix the update to the kernel, which was actually fine except it did not rebuild modules after updating


I can't help you with the rollback.

More techie users will need address this.

@Stevo? @fehlix?

There are many ways to post system info, which do you prefer ?
I never said that I dont want to install a backup kernel. Obviously I want to, but I dont just want to install a random kernel.
Preferably I want to reinstall the following which is the kernel that was replaced by the defect kernel during the update:

1. As has already been explained, the problem with the 4.19.0-5 kernel was (past tense) that when it first rolled out in the upgrades, it failed to reinstall dkms kernel modules as it was supposed to do, and as all of the other kernels do. This issue was corrected over a week ago.

The problem affected desktop and laptop PCs that have Broadcom wifi drivers, that use ndiswrapper to load Windows wifi drivers, and/or that have Nvidia's GPU drivers installed instead of using the nouveau driver: primarily the non-free drivers provided by particular hardware or software companies are what were affected. (It probably also impacted users who had VirtualBox or Open VM though no one reported problems with these.) You may see a list of dkms modules by opening MX Package Installer on your server, clicking on the MX Stable Repo tab, and entering "dkms" (without the quotation marks) in the search field. You can then examine that list and determine if your server uses any of them (any installed ones will be greyed out, which is how you can tell.) If not, then even if you currently have the buggy version of the kernel from over a week ago installed your server shouldn't be affected when rebooted.

2. Someone already explained how to manually cause the reinstallation of all dkms modules for that kernel or whatever your current kernel is. This can be done without rebooting. Doing this will prevent the possibility of system problems due to dkms modules not being installed in your kernel so it should then survive a reboot if your server uses any dkms modules. There should be no need to roll back the kernel to an earlier version once all dkms kernel modules have been manually forced to (re)install. This was actually the solution that was given to those who had already updated to the buggy kernel then found that their Nvidia graphics or Broadcom wifi no longer worked after rebooting, when the dkms kernel modules for those failed to be automatically reinstalled after the new kernel was installed. Since the modules didn't get automatically reinstalled during the upgrade process it was simply a matter of issuing a manual command, once, to make them do so. Kernel modules only need to be installed into a kernel once, after that kernel is installed.

3. I already explained how you can lock down your current kernel to prevent it from being automatically upgraded, using Synaptic Package Manager. You can do the same thing for any other individual packages that you wish to "pin" at their current versions.

4. IT industry best practices are, among other things, not to enable automatic updating on production servers until those updates have first been tested by system administrators on an identical sandbox or test server to make sure they don't break anything. If your server's uptime is very important I would advise you to follow that practice as well. Another best practice is of course to have redundant backups, not only of any user data on the server but of its entire operating system as well. MX Snapshot is a good tool for backing up the OS with all installed software. To restore the server to the state it was in when the snapshot was created, sans the user data which would need to be backed up and restored separately, one would simply reinstall into existing partitions from the .iso file that MX Snapshot created.

5. If the concern is that another kernel in the future may have similar issues as the 4.19.0-5 one did (past tense) installing an alternate, known-stable LTS kernel so it will be available to boot with if needed is highly recommended. Just select the kernel that you want (the package name will begin with "linux-image") in MX Package Installer, check the checkbox at the left of it, and click Install. It's as simple as that. Should the current kernel fail to boot, simply reboot and select the backup kernel in the grub boot menu under advanced options. Or do what I do: run MX Boot Options, check the checkbox to the left of "Use flat menus (no submenus", then a list of all available kernels will be displayed in the main grub menu. If you also enable "Enable saving last boot choice" then grub will remember which kernel you selected and use it from then on until you select a different one.

6. If you choose not to do any of these things you may, if you wish, continue running as-is without rebooting until such time as a newer kernel becomes available (but how will you know that it won't cause problems on your server until you test it by (re)booting with that new kernel? That's why testing updates on a test server with identical hardware (especially CPUs, network and video cards), and not allowing untested updates to be automatically installed on production servers, is industry best practice as I said.)

7. If your server is highly mission-critical I would strongly suggest reconfiguring it with a commercial Linux server product from a company that offers paid support plans, guaranteed uptimes and optional remote or onsite system administration, such as RedHat and SuSE, rather than using a free desktop-oriented distribution with only volunteer community support available, as good as that support may be. Neither MX Linux nor Debian Stable for the desktop (not Manjaro, Ubuntu, Mint, Arch, etc.) offer any guarantees whatsoever that a future upgrade absolutely will never break your particular machine. If you require that level of uptime support there are companies that sell it. As the saying goes, "you get what you pay for."

I apologize if this may sound sarcastic or condescending to the OP or anyone else. I assure you, that is not my intention. I have a background in computer support in enterprise organizations and I frequently "rubbed elbows" with server, network and even mainframe administrators, through working on their desktop computers or attending meetings together, and I also have had some server administration experience in a professional setting so I know how these things work and am just telling it like it is in the business and government worlds.

I hope this information is useful and I wish you the best of luck.

zimbodel wrote: Sun Jun 30, 2019 3:36 am There are many ways to post system info, which do you prefer ?

In MX Tools run Quick System Info, then click paste in a new post (formatting is done for you already).

zimbodel wrote: Sun Jun 30, 2019 3:36 am There are many ways to post system info, which do you prefer ?...

If you go back to my earlier post which I am now linking to for the 2nd time: viewtopic.php?p=512349#p512349

and refer to paragraphs 1 and 2, I said:

I think zimbodel just wants a secondary kernel installed so that when he reboots - which he hasn't done since the 4.19 kernel issue - he has something that works if his 4.19 doesn't.

If I'm correct in my understanding, zimbodel, the first step is to paste the output of MX Tools >> Quick system info here so that we know your hardware specs.

If OP still needs help, "Quick System Info" is a good to start with.

It seems we just all talk past each other. Simple crucial questions are ignored.

Anyway, thanks for the help. I am convinced we all had good intentions, but we talk past each other.

As bad luck has it, there was a power failure last night past my ups lifetime for this rack of servers.
Excactly what I feared and tried to avoid.

Upon reboot this new kernel is a royal pain. Since the new kernel was running the server dies every 2 hours and a myriad of virtual machines dont work, bluetooth, jack you name it. Not worth it.

It doesnt matter and I dont need any more help as I will reinstall the automatic dd created backup that is a month old, and can recover user accounts for most recent files from user account backups. All I wanted was help to get the old kernel back by installing it and then reboot with this previous kernel, and presented all information to that effect. asqwerth did pay attention to the problem, thank you very much.

So all became irrelevant now as the new kernel is running and it is buggy and unpredictable and what I tried to prevent now happened.

Thanks all for helping.

You were asked to run MX Tools/Quick System Info on the server and post the results multiple times by multiple people so they could have the information they needed to give you specific instructions. You didn't.

You were told how to manually install the dkms modules to the kernel that the original 4.19.0-5 kernel's installation failed to automatically install, to prevent issues with apps and hardware not working after a reboot. You apparently didn't as you're saying that now virtual machines and other module-dependent apps aren't working.

You were told how to install a second known-good kernel, which could even have been the original 4.19.0-1 kernel that you wanted, and make it available in the grub boot menu so it would be available to boot with, and also how to pin it in Synaptic to prevent that kernel from being updated. You didn't.

You could have explored the options in MX Boot Options and seen how to set the default kernel (or asked in the forum), so that after installing a known-good one you could have set it as the default in case of an unattended unintentional reboot. You didn't.

You allowed untested back-ups to be automatically installed on a production machine without testing them first on another server to ensure that they didn't cause any system issues.

You allowed your server to perform an ungraceful power-loss shutdown while it was running, after the UPS battery expired, rather than setting up a system where you'd automatically be paged or otherwise notified as soon as the UPS took over so you could remote in and perform a graceful shutdown, closing all open processes and unmounting all filesystems.

You could still recover your server within five minutes with little or no loss of data by simply installing another kernel and rebooting at this point, yet you choose instead to restore from a month-old backup, with consequent downtime and loss of more-recent data. (Really? You only back up a production server monthly?)

The current issues you're having with your server are entirely on you, friend.