Page 1 of 1

APT preferences & debian-pinning

Posted: Fri Dec 27, 2024 3:26 am
by SilverX
I recently realized that very little thought is given to APT's inner configuration by most distributions. Many users have unintentionally rendered their systems unusable because APT, due to a lack of proper configuration, mistakenly identifies critical packages as unnecessary and removes them—leading to catastrophic consequences.

This insight came to me while working on improving Entropy Linux to prevent it from breaking after a full system upgrade or encountering major issues when installing packages with Debian testing repositories enabled. During this process, I discovered two critical configuration files for APT that can drastically improve system stability:

/etc/apt/preferences
/etc/apt/preferences.d/debian-pinning
With these configurations properly set, APT transforms from a system prone to sudden failure into a robust package manager capable of handling multiple repository types (e.g., stable, testing, unstable, MX) simultaneously. It mitigates risks such as conflicting package versions or breaking crucial components like glibc6 during a full upgrade.

This improvement isn't hypothetical—I’ve experienced system-breaking updates on MX Linux and other distributions when attempting a simple apt update && apt upgrade. Such issues were especially frequent when a significant number of packages needed upgrading due to the lack of recent ISOs with updated packages.

Since implementing these configurations for Entropy, I’ve conducted numerous full upgrades with testing repositories enabled at all times. Additionally, I can now seamlessly specify which repository to use for individual packages using the -t flag. For example: sudo apt install -t testing <package>
This allows me to install newer versions from testing without hassle or risk. The difference is remarkable, and it’s a change I believe could benefit many others.

> /etc/apt/preferences

Code: Select all

# High priority for Debian Stable (Bookworm)
Package: *
Pin: release a=bookworm
Pin-Priority: 1001

# Security updates for Stable (Bookworm) with the same high priority
Package: *
Pin: release n=bookworm-security
Pin-Priority: 1001

# Medium priority for Debian Testing (Trixie)
Package: *
Pin: release a=trixie
Pin-Priority: 750

# Low priority for Debian Testing (to prevent auto-installation)
Package: *
Pin: release a=testing
Pin-Priority: 50

# Very low priority for Debian Unstable (Sid)
Package: *
Pin: release a=unstable
Pin-Priority: 1

# Extremely low priority for Debian Experimental
Package: *
Pin: release a=experimental
Pin-Priority: -1

# Medium priority for Debian Backports (Bookworm-Backports)
Package: *
Pin: release a=bookworm-backports
Pin-Priority: 500

# High priority for MX Linux Repositories (Main)
Package: *
Pin: origin nl.mxrepo.com
Pin-Priority: 1002
And
> /etc/apt/preferences.d/debian-pinning

Code: Select all

# High priority for Debian Stable (Bookworm)
Package: *
Pin: release o=Debian, a=bookworm
Pin-Priority: 1001

# High priority for Debian Stable Updates (Bookworm Updates)
Package: *
Pin: release o=Debian, a=bookworm-updates
Pin-Priority: 1001

# High priority for Debian Stable Security Updates (Bookworm Security)
Package: *
Pin: release o=Debian, n=bookworm-security
Pin-Priority: 1001

# Medium priority for Debian Testing (Trixie)
Package: *
Pin: release o=Debian, a=trixie
Pin-Priority: 750

# Low priority for Debian Testing (general)
Package: *
Pin: release o=Debian, a=testing
Pin-Priority: 400

# Very low priority for Debian Unstable (Sid)
Package: *
Pin: release o=Debian, a=unstable
Pin-Priority: 100

# Extremely low priority for Debian Experimental
Package: *
Pin: release o=Debian, a=experimental
Pin-Priority: 1

# Medium priority for Debian Backports (Bookworm Backports)
Package: *
Pin: release o=Debian, a=bookworm-backports
Pin-Priority: 500

# High priority for MX Linux Repositories (Main)
Package: *
Pin: origin nl.mxrepo.com
Pin-Priority: 1002
It's surprising that no one has thought to include this in their default distro configuration. If the omission is intentional for reasons such as minimalism, simplicity, or promoting user responsibility, these arguments no longer hold up in practice. Including this configuration requires only 1,949 bytes of storage, and its sole impact—whether intentional or not—is enabling the proper functionality of APT as originally designed. This prevents the package manager from inadvertently entering a "self-destruct" mode due to hidden or sudden issues.

It’s reasonable to assume that most users are sufficiently advanced while maintaining the principle of user responsibility for their systems. Common counterarguments, such as "it only happens if you mess with it," or "testing repositories are inherently unstable," are familiar refrains from staunch advocates of the traditional "vanilla" Debian approach.

In practice, however, using testing and unstable repositories while installing packages from these sources has significantly improved the stability of APT in my system. It has made Entropy Linux more resilient to issues than Debian, MX Linux, or any other Debian-based distro I’ve encountered. Astonishingly, less than 2 KB of carefully considered configuration can address one of Debian’s most significant issue and protect users from learning painful lessons the hard way.

This approach benefits everyone, particularly less experienced users who are prone to experimenting with APT—whether by attempting to install newer packages, running outdated or unverified commands from the internet, or inadvertently breaking their system. These scenarios are common and will continue to happen.

So, why not include these by default? :confused:

Re: APT preferences & debian-pinning

Posted: Fri Dec 27, 2024 4:08 am
by fehlix
SilverX wrote: Fri Dec 27, 2024 3:26 am This improvement isn't hypothetical—I’ve experienced system-breaking updates on MX Linux ...
We do strongly recommend not to adding randomly other repos, from other the distributions.
If you do, you have to prepare something like you propose, in order to "try" avoid breaking the system and keeping and taking the pieces home.

Re: APT preferences & debian-pinning

Posted: Fri Dec 27, 2024 4:38 am
by SilverX
Actually, MX broke on me few times on pretty much fresh install while doing full apt upgrade with +300 upgradable packages, with zero involvment from my side, nor even touching repos.
And by other repos, I mean debian sid and trixie.

Don't get me wrong, MX Is amazing distibution, that I still have special place in my heart for!

Although facts are it would be that tiny bit better with APT config.
Regardless if you'll use what I provided, or will write one yourself, I strongly reccomend doing so.
I can tell both from research and personal experience it does matter if you configure packages, especially as important as apt is.

Re: APT preferences & debian-pinning

Posted: Fri Dec 27, 2024 5:00 am
by asqwerth
SilverX wrote: Fri Dec 27, 2024 4:38 am Actually, MX broke on me few times on pretty much fresh install while doing full apt upgrade with +300 upgradable packages, with zero involvment from my side, nor even touching repos.
And by other repos, I mean debian sid and trixie.

Don't get me wrong, MX Is amazing distibution, that I still have special place in my heart for!

Although facts are it would be that tiny bit better with APT config.
Regardless if you'll use what I provided, or will write one yourself, I strongly reccomend doing so.
I can tell both from research and personal experience it does matter if you configure packages, especially as important as apt is.
Without terminal output I'm a bit skeptical about there having been 300+ (!) upgradable packages on a vanilla install without the adding of other repos. I'm not sure what you mean by "without touching repos". Does that mean you did add these other repos or perhaps manually installed deb files, but did not tweak the apt preferences?

The way MX wishes to operate is to stay binary compatible with Debian Stable, so no adding of Debian Testing or Sid repos. If newer packages are needed from Testing and Sid, our packaging team backports the newer packages to be compatible with Debian Stable.

I notice you keep namedropping that Debian distro in your posts. Not sure if it's your own project or what. Perhaps you don't mind that distro install being a FrankenDebian, and perhaps you are adept enough to sort things out on your own install, but MX is specifically Debian Stable-based.

Re: APT preferences & debian-pinning  [Solved]

Posted: Fri Dec 27, 2024 5:55 am
by DukeComposed
asqwerth wrote: Fri Dec 27, 2024 5:00 am Without terminal output I'm a bit skeptical about there having been 300+ (!) upgradable packages on a vanilla install without the adding of other repos.
On a lark, I booted an MX-23.2 ISO I had laying around and let MX Updater/apt-notifier check to see how behind it was. It let me know there were 383 new updates to apply. (337 to upgrade, 46 new packages to install, and 1 to remove.) 23.2 is about 11 months out of date, which isn't unheard of if someone intends to install an existing ISO they have and patch it to 23.4.
asqwerth wrote: Fri Dec 27, 2024 5:00 am Perhaps you don't mind that distro install being a FrankenDebian, and perhaps you are adept enough to sort things out on your own install, but MX is specifically Debian Stable-based.
What stood out to me is the statement "MX broke on me few times on pretty much fresh install while doing full apt upgrade" which means running an "apt full-upgrade"/"apt-get dist-upgrade" and hoping for the best. From man apt-get(8), which of course we all have memorized:

Code: Select all

dist-upgrade in addition to performing the function of upgrade, also intelligently
handles changing dependencies with new versions of packages; apt-get has a "smart"
conflict resolution system, and it will attempt to upgrade the most important packages
at the expense of less important ones if necessary. The dist-upgrade command may
therefore remove some packages.
dist-upgrade does exactly what it says on the tin: it will prioritize the big stuff at the risk of the little stuff, and delete things in the process if it sees a conflict. This matches what OP describes, but it's not something I've encountered in my years of upgrading MX, in part because of the Debian policy of keeping the same libraries and utilities for the lifetime of a release. Big breaking changes are typically put into an upcoming Debian release, so when I say "upgrading MX" I mean only running "apt-get upgrade". MX-21 installs stay at MX-21. I don't edit the apt sources to try to turn an MX-21 machine into MX-23, which dist-upgrade will try to do if you let it. Part of me thinks that when OP says "And by other repos, I mean debian sid and trixie", that means trying to dist-upgrade an old MX major release to a new one.

Re: APT preferences & debian-pinning

Posted: Fri Dec 27, 2024 6:10 am
by asqwerth
Mx-updater uses apt full-upgrade by default. I've not had any issues.

I guess 300+ updates is possible for an old iso. On the other hand, I've waited almost a year before upgrading mx on my dad's netbook and my old laptop in the past without issue. I've never had conflict issues with Debian stable even if I've not updated my system for a long time. I always use apt full-upgrade.

Re: APT preferences & debian-pinning

Posted: Fri Dec 27, 2024 6:32 am
by anticapitalista
Broken/un-updated package mirrors may also cause breakage.
Whenever there is a big upgrade, I always check that the default repos are working as they should.

Re: APT preferences & debian-pinning

Posted: Fri Dec 27, 2024 7:59 am
by fehlix
SilverX wrote: Fri Dec 27, 2024 4:38 am Actually, MX broke on me few times on pretty much fresh install while doing full apt upgrade with +300 upgradable packages, with zero involvment from my side, nor even touching repos.
So how many 3trd party, or ubuntu-like "foreign" deb-package have you installed, before the system broke?
Note: Even one innocent looking package, which even may work for a while, can cause a system to break - suddenly - after a big system upgrade.

Re: APT preferences & debian-pinning

Posted: Fri Dec 27, 2024 8:21 am
by asqwerth
@DukeComposed

It should be noted that apt-upgrade will only install updates that don't remove other packages or that don't install additional packages.

So if you used apt upgrade instead of apt full-upgrade, you won't pull in any new mx tools that are added in the course of the life of the release, nor will it install any upgraded packages that need new libraries /dependencies to be installed.

MX-updater has been apt full-upgrade or apt-get dist-upgrade all this time.

Re: APT preferences & debian-pinning

Posted: Fri Dec 27, 2024 8:25 am
by fehlix
asqwerth wrote: Fri Dec 27, 2024 8:21 am @DukeComposed

It should be noted that apt-upgrade will only install updates that don't remove other packages or that don't install additional packages.

So if you used apt upgrade instead of apt full-upgrade, you won't pull in any new mx tools that are added in the course of the life of the release, nor will it install any upgraded packages that need new libraries /dependencies to be installed.

MX-updater has been apt full-upgrade or apt-get dist-upgrade all this time.
Good, you mentioned that. The option "dist-upgrade" is certainly a misnomer, they replaced it later with "full-upgrade" (though keeping the old option name). Maybe they missed to change "upgrade" to something like "partial-upgrade".

Re: APT preferences & debian-pinning

Posted: Fri Dec 27, 2024 11:35 am
by DukeComposed
asqwerth wrote: Fri Dec 27, 2024 8:21 am It should be noted that apt-upgrade will only install updates that don't remove other packages or that don't install additional packages.

So if you used apt upgrade instead of apt full-upgrade, you won't pull in any new mx tools that are added in the course of the life of the release, nor will it install any upgraded packages that need new libraries /dependencies
I'm OK with that.

Re: APT preferences & debian-pinning

Posted: Fri Dec 27, 2024 5:46 pm
by SilverX
Sorry for late reply.
DukeComposed wrote: Fri Dec 27, 2024 5:55 am What stood out to me is the statement "MX broke on me few times on pretty much fresh install while doing full apt upgrade" which means running an "apt full-upgrade"/"apt-get dist-upgrade" and hoping for the best. From man apt-get(8), which of course we all have memorized:
- Well, yes, basically.
fehlix wrote: Fri Dec 27, 2024 7:59 am
SilverX wrote: Fri Dec 27, 2024 4:38 am Actually, MX broke on me few times on pretty much fresh install while doing full apt upgrade with +300 upgradable packages, with zero involvment from my side, nor even touching repos.
So how many 3trd party, or ubuntu-like "foreign" deb-package have you installed, before the system broke?
Note: Even one innocent looking package, which even may work for a while, can cause a system to break - suddenly - after a big system upgrade.
- Not really a frankendebian in my opinion, as long as it's stable, and performs well, while always being able to upgrade or downgrade most packages (most as few i compiled manually by hand as a optimal solution before fully automatic source build script is in place.
At this point (after dozen fails and months of work and tests), my distro is at peak stability a debian could be, few source built, and few other packages from sid / trixie, had so far no issues at all, it simply does what it's meant to do, and everything works and performs well, even under drastic conditions in stress tests. So far, not a single package broke since I implemented previously mentioned configs.
Stress tests included (but not limited to):
- Reinstalling same package, on same, and different version from different repo
- Downgrading packages and their dependencies significantly (by many versions)
- Upgrading everything (except glibc6*) to both testing and unstable if available.
So far, nothing even scratched it, worst issue so far i had with configs getting overriden by new version (which is not an actual issue at all, i just remade my configs properly, and it was and still is like nothing special ever happened)
Well, I had my fair share of making a broken hybrid system only by my own actions, Id say I've broke it in every possible way already. That's why Im confident in what I believe.
I might've not explained myself well, let me clarify what I mean by all that.
- Mx broke on me in those very rare and specific cases @DukeComposed explained, which is pretty much user error in less than optimal conditions. (Which is not reccomended, but does eventually happen one way or another)
MX Is very stable and super reliable, especially compared to some other debians, and MX repos are great! There is nothing wrong particularly with MX, don't get me wrong, as I agree MX Is exceptionally well designed!

What im trying to say, is Debian, is inherently created with few very specific, but oddly common ways average user can break his system, and while we all learned how to operate apt correctly, many users (I'd even say most), are not always familliar with complete works, functionality and every little detail of their system, like we do.
Believe me or not, but most Linux users are regular people, who are just fine with some basic knowledge they have to operate in day to day life.

What Im trying to do is make people realize adding just a few well thought out lines of config, you can instantly improve the actual linux debian experience for tousands of people, while not really changing anything, and having zero downsides, (there is no difference for you if you use apt the reccomended way anyway, while silently protecting you from consequences of silly mistakes and unholy experiments) There is zero difference if you don't have repos like trixie / sid added while configured in preferences file, so apart from ideological reasons, there are no reasonable arguments against going a bit out of our ways, just to include something that not everyone might use or even realize exist in everyday use. But something that has no downsides, is totalled to ~50 lines of text, (2kb), while providing significant improvements for many people,
from everyone who wanted to see what are those other debian repos all about, to everyone making simple user error (which is guaranteed to happen to everyone at least once), to people straight out doing things strictly forbidden in Debian bible.

I don't see any logical reason someone might not want to include something like that, just for the sake of regular everyday users, so someone's future silly mistake or mishap might not end up destroying their systyems in one way or another.
I mean, with preferences configured, you naturally still can kill any debian by forcing criticals to update from specific repo, and / or specific version, but that is not something you can ever do by accident, misfortune or curiosity, you have to be really specific to do any harm to a Debian.
(And while I said both MX, and Debian, I simply mean all Debians, as we all use same apt, with same rules, logic behind it, and mostly same repos. When I specifically mentioned that MX broke on me in that specific situation, I meant more like "Even as flawless and well designed system as MX, Is not immune to, as its something all debian based distros inhered day they were born")

You know what? Funny enough, but Linus Torvalds once said, he doesnt use Debian, because it is too confusing for him.
I think I understood what he meant by that.

Re: APT preferences & debian-pinning

Posted: Fri Dec 27, 2024 11:07 pm
by DukeComposed
SilverX wrote: Fri Dec 27, 2024 5:46 pm Funny enough, but Linus Torvalds once said, he doesnt use Debian, because it is too confusing for him.
I think I understood what he meant by that.
Linus doesn't use Debian because Fedora once let him change the time zone setting without needing the root password. Now he steadfastly refuses to learn a different OS because it would be more work than sticking with what he knows, and that includes having to give free tech support to his family. He's been pretty clear about this in the Q&As he's given.

Re: APT preferences & debian-pinning

Posted: Sat Dec 28, 2024 2:51 am
by asqwerth
DukeComposed wrote: Fri Dec 27, 2024 11:35 am
asqwerth wrote: Fri Dec 27, 2024 8:21 am It should be noted that apt-upgrade will only install updates that don't remove other packages or that don't install additional packages.

So if you used apt upgrade instead of apt full-upgrade, you won't pull in any new mx tools that are added in the course of the life of the release, nor will it install any upgraded packages that need new libraries /dependencies
I'm OK with that.
It's fine if you are ok with that. My point is that the dist-upgrade and full-upgrade commands are not simply for the purposes of upgrading to a new distro release of Debian after you change the repo sources. It's for more than that even if you stick to the default Debian and MX repos.

The apt full-upgrade command is default for MX-updater because new MX tools and other tweaks or patches are sometimes added, and also because MX backports of newer versions of packages may find its way into the main repos after some testing, and these newer versions sometimes require additional dependencies to be installed as well.