antiX live usb issue

Message
Author
User avatar
BitJam
Developer
Posts: 2303
Joined: Sat Aug 22, 2009 11:36 pm

Re: antiX live usb issue

#121 Post by BitJam »

TL;DR: I really think the problem is at least partly caused by the usb stick perhaps in combination with certain hardware. If you can create a live-usb stick that reliably fails in this strange way (especially the lz4 error) when booted an a variety of hardware, especially a newish machine, then I'd love to get you to send one to me so I can see it fail here and examine the failure in great detail.

But even if we can pin the problem down to a booting on a particular machine or with a particular live-usb or both in combination, I think I need to relent now and change it so we only try fuseiso if normal mounting fails. I still want to wait and see if anyone else reports a similar problem but I will post an easy solution for you that should fix all of the versions of live-usb-maker for you and won't go away when things get updated.

I'm sorry to have put you through so much bother. I'm really glad we were able to fix some of the fragmentation problems due to your help.

If you can create a live-usb that reliably fails on many different machines due to not using --force=nofuse then I would like to arrange to buy one from you or trade for one.


Way too many notes!
rs55 wrote: Thu Mar 28, 2019 5:12 pm dmesg : FAT-fs (sdb2): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
( thats the only line that seemed like an error msg)
uname -r : 4.20.12-antix.1-amd64-snp

(I dont know how to copy all the messages from dmesg to paste here- its on another laptop)
That's fine. So there was no lz4 message in the log. Good to know. So the original error message was what? Was it "Error: mount: mounting /dev/loop0 on live/linux failed: Invalid argument"?

It is helpful if you can gather the information from one incident all together: the kernel, the original error message, the extra dmesg error message (if any), and maybe even the session from the LUM log file.

You seem to be having a variety of different problems. Just from this thread there was the first the issue with reading the vmlinuz file. Then there was the lz4 error when mounting linuxfs and now you have yet a different kind of error when we try to mount linuxfs.

I still want to see the output of "uname -r" when dmesg tells you there is an lz4 error. If the kernel version supports lz4 then that would be a very bizarre situation. Very bizarre.

Do these errors still only happen when you make a live-usb on one particular machine and not any others?

Do the bad live-usbs fail to boot on different machines or just one?

Do the problems all go away when you use --force=nofuse?

Do they go away when you use brand of usb stick?

If you happen to be using SanDisk Ultra Fit, then stop doing that, at least for live-usbs. We know they cause a variety of strange hardware problems.
rs55 wrote: Thu Mar 28, 2019 5:33 pmwhere would I find the log file for the failed lum ?
Almost all log files are under /var/log/. This is no exception. The log file is at /var/log/live-usb-maker.log. The most recent recent session gets added to the bottom of the file. There is a ton of information for each session about when it was made and what the distro was made, what the command line options were and so on so it should not be too hard to find a particular session. Of course the most recent session is always at the bottom of the file.

I'm fairly confident the vmlinuz file is not fragmented. The log file would tell us for sure. It may be good to continue to use "checkmd5" to be sure there is no file corruption but you have never found any. Passing the md5 test tells us that 3 key fails are perfectly intact and it tells us further that Linux can read them perfectly.

Since the vmlinuz file loads and is likely not fragmented then I don't think there is a problem with the kernel. It would be an astronomical coincidence that the kernel was somehow corrupted and that corruption only affected attempts to mount the linuxfs file (now in two different ways!).

But I don't understand why Linux can read the file when we do the md5 test but then has trouble reading it (or something???) when we try to mount it. Normally, I would expect to see IO errors in the dmesg output when there is a hardware problem. But very occasionally, the hardware can give you the wrong numbers without throwing an error. I've seen it just once before.

There seems no doubt that you have a system that generates much more fragmentation than most systems when fuseiso is used. Yet we fixed the problem with reading vmlinuz and the md5 check tells us Linux can read the 3 files fine despite fragmentation and they are all intact.

My best guess now is there is a problem with the system you are booting or with the usb-stick or both that is sensitive in some strange way to the extra fragmentation. For example the SanDisk Ultra Fit seems to gulp too much power in some situations that can cause hardware glitches on some systems. This could be accentuated on a older system because the electrolytic capacitors that are used to prevent such power glitches are usually the first part to wear out.
"The first principle is that you must not fool yourself -- and you are the easiest person to fool."

-- Richard Feynman

rs55
Posts: 273
Joined: Sun Feb 24, 2019 3:24 pm

Re: antiX live usb issue

#122 Post by rs55 »

rs55 wrote: Thu Mar 28, 2019 6:17 pm
rs55 wrote: Thu Mar 28, 2019 6:14 pm
dolphin_oracle wrote: Thu Mar 28, 2019 6:05 pm /var/log/live-usb-maker.log
see attached. Also - to confirm: uname -r : 4.20.12-antix.1-amd64-smp
Also: with --force=nofuse , the usb boots up fine
This log is for the failed usb creation ( without the --nofuse)live usb log.txt
The mystery ( at least to me) is - this exact same kernel, running this exact same version of lum, works fine from the antix17.4 install ( Control Center - gui).
Ran another experiment - this time using the same exact iso. ( its a 7.8 GB iso). Both MX and Antix fail. So the issue is with this particular iso. It works fine with --nofuse. But fails otherwise.
Is it the size? ( my other iso, the antix iso is 1.7 GB - and works fine without --nofuse).

User avatar
dolphin_oracle
Developer
Posts: 22219
Joined: Sun Dec 16, 2007 12:17 pm

Re: antiX live usb issue

#123 Post by dolphin_oracle »

The iso size is important clue. Perhaps fuseiso has a problem with isos outside a given size. Dvd would be around 4.2 GB max
http://www.youtube.com/runwiththedolphin
lenovo ThinkPad X1 Extreme Gen 4 - MX-23
FYI: mx "test" repo is not the same thing as debian testing repo.

User avatar
BitJam
Developer
Posts: 2303
Joined: Sat Aug 22, 2009 11:36 pm

Re: antiX live usb issue

#124 Post by BitJam »

rs55 wrote: Thu Mar 28, 2019 7:20 pmRan another experiment - this time using the same exact iso. ( its a 7.8 GB iso). Both MX and Antix fail. So the issue is with this particular iso. It works fine with --nofuse. But fails otherwise.
Is it the size? ( my other iso, the antix iso is 1.7 GB - and works fine without --nofuse).
Yes! It could be the size. One more thing to try is to mount the broken live-usb and then run:

Code: Select all

sudo e4defrag $MOUNT_POINT/antiX/linuxfs
This will take a while to run but it will try to defrag the linuxfs file.

I still think there is a problem with the usb stick and or the hardware. If I'm wrong and you can get different brands of usb-sticks to fail on a variety of hardware, especially newish hardware (less than 4 years old) then I would like to get one of those sticks from you.
dolphin_oracle wrote: Thu Mar 28, 2019 7:39 pm The iso size is important clue. Perhaps fuseiso has a problem with isos outside a given size. Dvd would be around 4.2 GB max
I don't think so because the md5 check said the linuxfs file was fine. We have never seen fuseiso give us the wrong files. The md5 checks and diffs always said everything was identical. With a larger file there will be more fragmentation and perhaps greater jumps in where the fragments are and I believe this is triggering a problem on the hardware rs55 is using. Remember rs55 had a problem even when the vmlinuz file was slightly fragmented.
"The first principle is that you must not fool yourself -- and you are the easiest person to fool."

-- Richard Feynman

rs55
Posts: 273
Joined: Sun Feb 24, 2019 3:24 pm

Re: antiX live usb issue

#125 Post by rs55 »

BitJam wrote: Thu Mar 28, 2019 7:07 pm TL;DR: I really think the problem is at least partly caused by the usb stick perhaps in combination with certain hardware. If you can create a live-usb stick that reliably fails in this strange way (especially the lz4 error) when booted an a variety of hardware, especially a newish machine, then I'd love to get you to send one to me so I can see it fail here and examine the failure in great detail.

But even if we can pin the problem down to a booting on a particular machine or with a particular live-usb or both in combination, I think I need to relent now and change it so we only try fuseiso if normal mounting fails. I still want to wait and see if anyone else reports a similar problem but I will post an easy solution for you that should fix all of the versions of live-usb-maker for you and won't go away when things get updated.

I'm sorry to have put you through so much bother. I'm really glad we were able to fix some of the fragmentation problems due to your help.

If you can create a live-usb that reliably fails on many different machines due to not using --force=nofuse then I would like to arrange to buy one from you or trade for one.


Way too many notes!
rs55 wrote: Thu Mar 28, 2019 5:12 pm dmesg : FAT-fs (sdb2): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case sensitive!
( thats the only line that seemed like an error msg)
uname -r : 4.20.12-antix.1-amd64-snp

(I dont know how to copy all the messages from dmesg to paste here- its on another laptop)
That's fine. So there was no lz4 message in the log. Good to know. So the original error message was what? Was it "Error: mount: mounting /dev/loop0 on live/linux failed: Invalid argument"?

It is helpful if you can gather the information from one incident all together: the kernel, the original error message, the extra dmesg error message (if any), and maybe even the session from the LUM log file.

You seem to be having a variety of different problems. Just from this thread there was the first the issue with reading the vmlinuz file. Then there was the lz4 error when mounting linuxfs and now you have yet a different kind of error when we try to mount linuxfs.

I still want to see the output of "uname -r" when dmesg tells you there is an lz4 error. If the kernel version supports lz4 then that would be a very bizarre situation. Very bizarre.

Do these errors still only happen when you make a live-usb on one particular machine and not any others?

Do the bad live-usbs fail to boot on different machines or just one?

Do the problems all go away when you use --force=nofuse?

Do they go away when you use brand of usb stick?

If you happen to be using SanDisk Ultra Fit, then stop doing that, at least for live-usbs. We know they cause a variety of strange hardware problems.
rs55 wrote: Thu Mar 28, 2019 5:33 pmwhere would I find the log file for the failed lum ?
Almost all log files are under /var/log/. This is no exception. The log file is at /var/log/live-usb-maker.log. The most recent recent session gets added to the bottom of the file. There is a ton of information for each session about when it was made and what the distro was made, what the command line options were and so on so it should not be too hard to find a particular session. Of course the most recent session is always at the bottom of the file.

I'm fairly confident the vmlinuz file is not fragmented. The log file would tell us for sure. It may be good to continue to use "checkmd5" to be sure there is no file corruption but you have never found any. Passing the md5 test tells us that 3 key fails are perfectly intact and it tells us further that Linux can read them perfectly.

Since the vmlinuz file loads and is likely not fragmented then I don't think there is a problem with the kernel. It would be an astronomical coincidence that the kernel was somehow corrupted and that corruption only affected attempts to mount the linuxfs file (now in two different ways!).

But I don't understand why Linux can read the file when we do the md5 test but then has trouble reading it (or something???) when we try to mount it. Normally, I would expect to see IO errors in the dmesg output when there is a hardware problem. But very occasionally, the hardware can give you the wrong numbers without throwing an error. I've seen it just once before.

There seems no doubt that you have a system that generates much more fragmentation than most systems when fuseiso is used. Yet we fixed the problem with reading vmlinuz and the md5 check tells us Linux can read the 3 files fine despite fragmentation and they are all intact.

My best guess now is there is a problem with the system you are booting or with the usb-stick or both that is sensitive in some strange way to the extra fragmentation. For example the SanDisk Ultra Fit seems to gulp too much power in some situations that can cause hardware glitches on some systems. This could be accentuated on a older system because the electrolytic capacitors that are used to prevent such power glitches are usually the first part to wear out.
lets back up a bit:
- I already posted earlier that the lz4 error yesterday was when I was using kernel 4.9. That is no longer an issue with kernel 4.20
- I am confident its not the flash drive. i just tried a brand new Samsung flash drive - same error upon boot.
- The error is also repeatable from both the Antix system and the MX system - no difference.
- There is no problem when I use my 1.7GB iso image.
- The problem occurs when I use my 7.8 GB iso image.
- The problem with the usb boot with the 7.8 GB image goes away if I use LUM with --force==nofuse
- I was not having a single issue until the latest lum upgrade( which is using fuseiso), so I have had to revert back and stop all upgrades.
- i would like to upgrade normally at some point - so any advice from you would be helpful. Or maybe I have to be resigned to always running lum from the terminal and using --force=nofuse.

User avatar
BitJam
Developer
Posts: 2303
Joined: Sat Aug 22, 2009 11:36 pm

Re: antiX live usb issue

#126 Post by BitJam »

rs55 wrote: Thu Mar 28, 2019 7:20 pmRan another experiment - this time using the same exact iso. ( its a 7.8 GB iso). Both MX and Antix fail. So the issue is with this particular iso. It works fine with --nofuse. But fails otherwise.
Is it the size? ( my other iso, the antix iso is 1.7 GB - and works fine without --nofuse).
I quoted to get your attention. As promised, I've attached a fix for you. If you put the tarball in your home directory then you can install it like this:

Code: Select all

cd /usr/local
sudo tar xzf ~/rs55-lum-fix-01.tgz
You can test if it is installed correctly by running:

Code: Select all

live-usb-maker --version
It should show you something like:

Code: Select all

      live-usb-maker version 2.30.00 (Sun Mar 24 12:37:30 MDT 2019)
     cli-shell-utils version 2.31.00 (Thu Mar 28 17:50:32 MDT 2019)
The crucial thing is that the lib is version 2.31.00.

The tarball contains a cli-shell-utils/ directory that contains the modified cli-shell-utils.bash library and a few helper apps. Oh, translations will be disabled. If you need that then I can give you a bigger tarball.

The normal version of live-usb-maker that is used by the guis and when you call "live-usb-maker" on the command line lives in /usr/local/bin. It will look in /usr/local/cli-shell-utils/ first for the library and if it finds it there it won't look in the standard location.

If you want to continue to experiment with this wacky fragmentation issues then copy /usr/local/bin/live-usb-maker to your home directory and run it as:

Code: Select all

sudo ~/live-usb-maker
You can test this with the --version option and it should show you the older library version.

Again, if you install this fix and copy live-usb-maker to your home directory then "live-usb-maker" and "/usr/local/bin/live-usb-maker" will use the fixed library while "~/live-usb-maker" will use the standard library that tries "fuseiso" first.

HTH
You do not have the required permissions to view the files attached to this post.
"The first principle is that you must not fool yourself -- and you are the easiest person to fool."

-- Richard Feynman

rs55
Posts: 273
Joined: Sun Feb 24, 2019 3:24 pm

Re: antiX live usb issue

#127 Post by rs55 »

You may not get many others who complain of this. Mostly because if the size of the iso is the issue, most people probably have small iso files <2GB. ( more than enough for the standard suite of apps - libreoffice, firefox etc).
I happen to have a lot of heavy programs loaded like Ananconda, tensorflow , rstudio etc, which makes the iso large.

User avatar
BitJam
Developer
Posts: 2303
Joined: Sat Aug 22, 2009 11:36 pm

Re: antiX live usb issue

#128 Post by BitJam »

rs55 wrote: Thu Mar 28, 2019 7:53 pmlets back up a bit:
- I already posted earlier that the lz4 error yesterday was when I was using kernel 4.9. That is no longer an issue with kernel 4.20
- I am confident its not the flash drive. i just tried a brand new Samsung flash drive - same error upon boot.
- The error is also repeatable from both the Antix system and the MX system - no difference.
- There is no problem when I use my 1.7GB iso image.
- The problem occurs when I use my 7.8 GB iso image.
- The problem with the usb boot with the 7.8 GB image goes away if I use LUM with --force==nofuse
- I was not having a single issue until the latest lum upgrade( which is using fuseiso), so I have had to revert back and stop all upgrades.
- i would like to upgrade normally at some point - so any advice from you would be helpful. Or maybe I have to be resigned to always running lum from the terminal and using --force=nofuse.
Thank you for the concise summary! I'm sorry if I was not following everything you said closely enough. I am buried in work to get antiX-19 (and then MX-19) out the door. At this rate I am about to start missing deadlines.

Anyway, the fix in my previous post should fix the problem for you and should survive updates.

If we can't replicate your problem using a large linuxfs and fuseiso then I would really like to buy or trade for that live-usb that you can get to fail reliably. Let's first see if we can replicate the problem and also please use "checkmd5" (if you haven't already). If checkmd5 fails then it is a bug in fuseiso, plain and simple. If it passes then the problem is more subtle.
"The first principle is that you must not fool yourself -- and you are the easiest person to fool."

-- Richard Feynman

User avatar
BitJam
Developer
Posts: 2303
Joined: Sat Aug 22, 2009 11:36 pm

Re: antiX live usb issue

#129 Post by BitJam »

YES! YOU ARE ABSOLUTELY RIGHT rs55! :number1: :number1:


You have found another problem with fuseiso. I made a 10G or so file and put it into an iso file and mounted it normally and with fuse iso. Here is the normal mount:

Code: Select all

-rw-r--r-- 1 juser juser 9.9G Mar 28 18:42 other.sqfs
Here is the mount with fuseiso:

Code: Select all

-rw-r--r-- 1 juser juser 4.0G Mar 29  2019 other.sqfs
-rw-r--r-- 1 juser juser 4.0G Mar 29  2019 other.sqfs
-rw-r--r-- 1 juser juser 4.0G Mar 29  2019 other.sqfs
Clearly we cannot use fuseiso with linuxfs files over 4.0G.

OTOH, The md5sums do not match. If you had done that simple test like I asked then it might have saved us all a lot of grief.
"The first principle is that you must not fool yourself -- and you are the easiest person to fool."

-- Richard Feynman

rs55
Posts: 273
Joined: Sun Feb 24, 2019 3:24 pm

Re: antiX live usb issue

#130 Post by rs55 »

BitJam wrote: Thu Mar 28, 2019 9:10 pm YES! YOU ARE ABSOLUTELY RIGHT rs55! :number1: :number1:


You have found another problem with fuseiso. I made a 10G or so file and put it into an iso file and mounted it normally and with fuse iso. Here is the normal mount:

Code: Select all

-rw-r--r-- 1 juser juser 9.9G Mar 28 18:42 other.sqfs
Here is the mount with fuseiso:

Code: Select all

-rw-r--r-- 1 juser juser 4.0G Mar 29  2019 other.sqfs
-rw-r--r-- 1 juser juser 4.0G Mar 29  2019 other.sqfs
-rw-r--r-- 1 juser juser 4.0G Mar 29  2019 other.sqfs
Clearly we cannot use fuseiso with linuxfs files over 4.0G.

OTOH, The md5sums do not match. If you had done that simple test like I asked then it might have saved us all a lot of grief.
Excellent Commander!!! So the issue all along was the large size ( 7.8GB ) of the troublesome iso. Thank you so much for the fix you provided - worked like a charm. I installed the fix first, then upgraded to the new lum. And the fix survived - which was my prayer! Brilliant!!
Again, thank you so much for taking time from your busy schedule to attend to this issue. Made my day! Sorry if I was not fully attentive to your debugging instructions - still very new to this linux business.
So - now that you know the size limitation - will you be fixing it - for the "normal version"? a 4GB+ iso would not seem all that unusual - that is if one is using the computer to do something other than web browsing !!

Post Reply

Return to “Software / Configuration”