Magic suggestions

Request new features that might make using Gentoox better.
Post Reply
boltronics
Newbie
Posts: 7
Joined: Sat Jan 08, 2005 4:06 am
Location: Hong Kong

Magic suggestions

Post by boltronics »

Errors/nitpicks I discovered in /bin/magic:
1. The use of the -r switch to the rm command when applied to a single file.
2. Line 421 states downloading magic.md5 failed, when patches.md5 failed.
3. I'd like to see cleanup() moved to /etc/functions.magic so it's not duplicated in the patches file.


Errors/nitpicks in current patches file:
1. The (compatibility) code will check for the functions.magic and patches.magic files, and assume they are fine if present. If they are an old version however, the patch wouldn't know since it doesn't perform an md5sum check.
2. Again, the use of the -r switch to the rm command when applied to single files.
3. Should use arguments to functions such as delDatabase, instead of exporting variables.
4. Never, ever, ever change my default editor from nano to vim! Unforgivable! It would be better to check if EDITOR is not present or undefined before making changes. Remember - Gentoo is about choice and a lot of us like to do things our own way. ;).
5. Gentoox user check assumes that the xbox admin wouldn't ever remove that user manually. It would be good to check /etc/passwd, and perhaps modify the GECOS field for the Gentoox user to say something like "Dependency for magic" so as to alert the unwary.
6. Looks relatively easy to circumvent the 5k/s download limit... probably should make that a server limitation rather than a client limitation (if possible).
7. Line 800 - `chmod 777 /bin/magic` looks... scary. Also similar issue for lines 1877 and 1933.
8. Isn't it possible to install mplayer without X using USE="-X"? Or in this circumstance does magic become irrelevant (since the patch is X-specific)?


Errors/nitpicks in the current xserver.b0x file:
1. Line 84 - chmod removes the suid bit, hence 'startx' fails to work for people who like to disble xdm.
2. Again, I don't like the /etc clutter.


I haven't been completly through the functions.magic file, or most of the b0x
files, so I can't comment on these yet.


Recommendations:
1. Move functions.magic to magic/functions and patches.magic to magic/patches. Also, these likely belong somewhere other than the /etc (configuration) directory since any manual editing will cause md5sum check by /bin/magic to fail and hence changes will be overridden.
2. Ditch the patches file compatibility code - it's probably too much duplication. If people are really running such ancient magic code and finally decide they want to use magic once more, have the old incompatible versions of magic detected and just point them to a URL with instructions on how to 'manually' download and install the new version.
4. Move /etc/gentoox_version to /etc/gentoox-release to be more inline with Gentoo standards.
5. Always ensure the EXTRAVERSION of the kernel Makefile matches the /usr/src/linux-... directory name to move inline with the Gentoo naming convention. The default kernel for 1.5Pro didn't seem to do this.
6. Move /etc/conf.d/oneshotmagicupdate to the same location as the new magic/ folder (mentioned in point 1).
7. Find a way for the admin to ditch the gentoox user - it shouldn't be essential, only optional.
8. You're probably going to seriously hate this suggestion, but the whole time I was reading your scripts I just kept thinking... this really goes against the Gentoo way of doing things. I realize that magic has evolved from copy and paste commands and patches, to b0x patches, to the b0x management system we now have, all probably without specifically being planned. Shallax, you often ask on the Gentoox main page what Gentoox users would like to see, and that's really commendable that you'll change directions with regards to magic if it'll make people happier.

However if I were to try and achieve the same goal that magic has, and I were to start over, I would look into a system that utilizes PORTDIR_OVERLAY more effectively. Perhaps if an official ebuild were broken (and I realize you have little power to fix this nowadays), you could have a tool (which is installed via ebuild, via the PORTDIR_OVERLAY directory) that checks your website for PORTAGE_OVERLAY updates (that are used to replace broken official ebuilds), and a working combination of package masks for files in /etc/portage. This way, users can still very much continue to do things 'the Gentoo way' and have a much clearer understanding of exactly what you intend to do to a Gentoox system without an end user having to read your source code. They may feel they are not in the dark anymore, as I once did.

As a side effect, it might encourage users to submit updates thus lifting the workload from your shoulders. As it stands at the moment, having a patch file run and then subsiquentely deleted requires some investigation for an end user to get ahold of. Correct me if I'm wrong, but it's not available on the website (although the b0x files are).

Looking through the portage.b0x file now, I see work has been started on something like this, but essentially it'd be really sweet to have a magic 'ebuild' that a user can chose to install, rather than have a modded-meta-distro. This would be already installed by default on your Stardust releases so the end user who doesn't care about any of this shouldn't notice much, if anything.


Final comments:
Very nice job. Without Gentoox, I wouldn't have bothered to purchase an Xbox in the first place. It's easy for me to sit back and complain, but you've battled on for longer than I would have bothered, and even maintained more frequent updates that I ever have for my personal projects.

Also, thanks for your recommendation (viewtopic.php?t=2467) of the Xenuim Ice. I can't believe what I was missing out on before. :)
ShALLaX
Site Admin
Posts: 1973
Joined: Sun Aug 10, 2003 9:25 pm
Location: England
Contact:

Post by ShALLaX »

This kinda belongs in the "requests" forum *moved*

Some good suggestions, but some very pedantic ones at that!

Magic:
1) Big deal - force of habit to type rm -rf - it doesnt cause any harm.
2) Fixed
3) Not possible because of the way a magic update works (it deletes functions.magic then downloads it again upon the next load of Magic... if the file isnt there, then the cleanup() routine isnt there... and we run into trouble if the user hits ctrl-c before its downloaded functions.magic).

Patches:
1) The files CANNOT be older, once a magic-update is done, it deletes these files an FORCES them to be redownloaded.
2) Big deal, this isnt an issue
3) Another non-issue
4) My patch system, my rules ;)
5) My distro, my rules. Dont touch the Gentoox user ;)
6) Its not _that_ easy, but its not foolproof... I realise this was an issue and was hoping no one would bother to look into it until I have my fix in place. I have coded my own HTTPd which will become the magic server. Once I've submitted it (its a University project) and had it graded, I'll add speed limiting and implement it.
7) Fixed (755).
8) I think by now USE="-X" is irrelevant.

Xorg:
1) Fixed - my bad.
2) Deal with it - thats what etc is for, IMO.

Part of the Gentoox philosophy is backwards compatibility. Regardless of duplication, I want this to be in.

gentoox_version... gentoox-release, another non-issue in my mind.

Im not moving to an overlay dir. I've invested too much time in Magic - sorry. Also my past working with Gentoo has made me determined not to integrate.

I get the feeling you think I want to be somehow standardised with Gentoo, when thats not really the case at all. If you want a standard "Gentoo" ... then download Gentoo-Xbox from http://www.gentoo.org.

Sorry for the disjointedness of this post, I was writing it quickly since I have to go to work!
The original Xbox adaptation of Gentoo
boltronics
Newbie
Posts: 7
Joined: Sat Jan 08, 2005 4:06 am
Location: Hong Kong

Post by boltronics »

Magic:
1. I know that the -r switch 'should' be a non-issue, but to me it's like a gun with the safety switched off. You state that backwards compatibility is important to you, but what if I decided to use a functions.magic directory to house the configuration files of my personal project? I do a magic, and BAM... I just lost my work. You think that's too unlikely? I freaked out at fist the first time I tried Gentoox because of all the references to 'stardust' - the name of my other computer on my network!
3. Ah - of course. I forgot about the CTRL+C option. Serves me right for posting at 1:30am.

Patches:
2. Still worries me. IIRC, there's no need for a recursive deletion in any of your scripts (apart from the overlay stuff), hence a s/rm -rf/rm -f/g could probably fix all of this.
3. Again, there is the possibility of overwriting something (in this case, a variable). You're assuming that people are always going to do everything your way and not the Gentoo way.
4) My patch system, my rules
Seriously, is there any reason why stardust can't just use nano as the default, and have magic not override any changes the user manually makes? I know I haven't looked everywhere, but it is super-frustrating!

5. I get that you don't want people to touch the gentoox user, but the point of having the user is not really clear. Perhaps for a first-time GNU/Linux user that wants to take a test drive on the Xbox this would be convenient. However I'm sure all other GNU/Linux users (especially Gentoo users) would already have a login name that they commonly use and thus not require or desire anything else.

It shouldn't be too hard to check if the user still exists in any code that depends on that user. And basically, AFAICT that should be limited to X and VNC (assuming that they are even installed)!

6. Well, I guess I won't submit a patch... but it'd be a matter of modifying /bin/magic to use sed to traverse through the patch file after downloading, and also using another temporary file to ensure that when /bin/magic is overridden, it will attempt to apply a patch (and use etc-update style prompting if it fails due to a 'real' update). You're lucky that nobody would bother anyway though since most files are very small.

8. Please clarify what you meant by this. You mean that you expect everyone will/does use X? The X flag is used for adding support for X11 (presumably the virtual package that either the depreciated XFree86 packages or the newer X.org packages and the like fulfills). Thus for many people it's still quite relevant.

Xorg:
2) Deal with it - thats what etc is for, IMO.
Well, try this:

Code: Select all

find /etc -type f | xargs file | grep -i "elf"
As you can see, practically nobody puts files of this type here. And there's a reason for this:
http://www.pathname.com/fhs/pub/fhs-2.3 ... FIGURATION

To quote the first requirement for the /etc directory:
No binaries may be located under /etc.
The backup of the Xorg file in /etc seems pointless anyway, since there was another backup in /usr/X11RC/bin/ (IIRC). As for the other files, to comply with the FHS-2.3 I believe functions.magic should be placed in /usr/share/magic/functions, and the patches.magic file should be placed under /var/lib/magic/patches.

All major distributions follow these. I've linked SuSEs structure for an example.
http://www.novell.com/partners/yes/linu ... ndard.html
Also my past working with Gentoo has made me determined not to integrate.
But wouldn't it make your life easier once it was initially hacked together? Or do you mean you are deliberately trying to not be compatible so they can't screw you around? Either way, I know that would take at least a few days to get this all set up, so I can understand why you don't think it would be worth it. I'm just curious.
I get the feeling you think I want to be somehow standardised with Gentoo, when thats not really the case at all.
Well kind of. I'm just assuming that for a lot of people it would be a lesser learning curve for people already using Gentoo on another computer. I definitely think following the FHS and LSB are always a good idea though. It would also be really nice to be able to use Gentoo tools to learn what magic is up to. eg. `qpkg -l magic` would make me feel a lot better, than having to worry about "Hmm... what's this Shallax up to today? I'd better go hack /bin/magic to save a copy of the patches file and not execute it to be safe. Then I can go and read a few thousand lines of source so I can establish where (hopefully) all of the files are placed around my system. That's why I previously suggested the /etc/magic folder - so at least I have a clue.

Also, somebody could one day crack your server while you're away, and replace the downloaded patch with `rm -rf / > /dev/null 2>&1` and corresponding md5, and all gentoox-boxen would disappear. It appears a disaster waiting to happen! OTOH, with emerge, I can preview (at least in abstract form) what will happen. This is sufficient, since the rsync server is typically different to a mirror server, and md5sums prove nothing has been modified. It's relatively secure.

I guess what I am saying is that there needs to be a way to give more power back to the user. Gentoo's emerge provides a lot of control, and hence my previous suggestion for integrating magic into emerge instead of working against it. Although this seems the obvious method, if there are problems with this route, then perhaps at least a /etc/magic/config file could be used to specify options such allowing the user to manually verify if they want their /etc/env.d/00basic file updated, etc. Novices don't even need to know it's there. Maybe you can even specify an option to have the newest patches updated (perhaps by CVS), and the admin can simply look at what's new and run them if appropriate. A more accurate change log would be super-helpful also. There are plenty of alternate ways to achieve the same goal.

How does that sound to you?
ShALLaX
Site Admin
Posts: 1973
Joined: Sun Aug 10, 2003 9:25 pm
Location: England
Contact:

Post by ShALLaX »

1) VERY unlikely - the file already exists there as functions.magic -- if people want to remove it to place their stuff there, they are asking for trouble.

3) ;)

Patches

2) BAH! Relax - In two years I have NEVER had any issues with this.

4) Fair play on the EDITOR issue, magic will only do it once now and then if the user changes it... magic wont touch it anymore.

5) The point of the user is that its safe to login as this user - it has preconfigured KDE/ XFCE environments, preconfigured onscreen keyboard layouts (see inside xbvset) the user MUST be there for the system to work correctly. If that user isnt there, there has to be some sort of way for a person to add a user on first boot, and I'm sure for those non-experienced Linux users, this could be an issue. What I can suggest is that you delete the user, but leave /home/gentoox as it is. This should be safe. If you want to maintain absolute compatibility, just set the "gentoox" user's shell to /bin/false - problem solved.

6) Anyone that respects the project wouldnt try to hack it. Problem is, you cant trust everyone.

8) Originally, the USE="-X" was to stop the pro version from installing X when mplayer was being installed. I dont know where you are actually seeing -X in the magic repo, because I dont see it anywhere (unless it comes preinstalled with -X?)


/etc... yes, naughty me putting ELFs in there. This really is a non-issue IMO. The binaries are not RUN from /etc... they are just put there to be copied when xbvset is run. "etc" as it suggests means "etcetera"... to me that means "anything that doesnt fit in the other directories" which I believe is the case here.

Its not a "screw you", its more of a "I dont want to be associated" thing.

Unfortunately, a lot of what youre saying requires nearly a total redesign of magic... and I neither have the time or patience to do it.

My solution to you would be... either use Gentoo-Xbox, or rm -rf /bin/magic (;)) and do everything yourself :p. Magic is a very selfish patch system indeed, and I dont hide that. It sets stuff up how I like it. My hope is that other people will also like it - if not, then dont use it and do it yourself - as mean as that sounds.
The original Xbox adaptation of Gentoo
boltronics
Newbie
Posts: 7
Joined: Sat Jan 08, 2005 4:06 am
Location: Hong Kong

Post by boltronics »

Magic:
1. Ah. I haven't tried the earlier versions of Gentoox, so I wasn't aware that those files would always be there. My bad.

Patches:
2. :)
4. Yay! :D
5. Very true that it can easily be disabled with a default /bin/false shell. I also understand that it could be very handy to have for a user who just installed Gentoox from stardust, and I'm probably just being too fussy. But what I was kind of getting at was that if they user is deleted, perhaps the script could say "Well this guy deleted the Gentoox user manaully, so although he probably knows jack about magic, he must know somethng about GNU/Linux and what he's doing". Then, it'll ignore any Gentoox user-specific commands. Maybe this will require re-working more code than I'm aware of though.
6. Yep.
8. I'm talking about line (aprox.) 1256 that states:

Code: Select all

# If xserver is not wanted, xvkbd, mplayer and kde cannot be installed
If USE="-X" was used, mplayer could be left as an option.
rm -rf /bin/magic
Very funny. ;)

Also, while I'm on a roll... I recall when I first installed Gentoox 1.5 Pro that there was a problem with the keyboard not working properly. The kernel seemed to have the default character encoding configured for UK, as did GNOME, /etc/rc.conf, X, and probably others that I can't thnk of. Maybe when installing Gentoox you could give the user the option of manually choosing their keyboard type from a selection of the 3 most common.

Is it going to be possible to boot a 2.6-series kernel soon? At the moment, I've only had success with the 2.4 series and a custom initrd script. If it's already possible, would you be able to provide a link to the relevant instructions (perhaps even under the FAQ section) since that would be very handy (even if fatx support is broken).

I've also noticed that connected bluetooth devices (I've tried two) always seem to crash the Gentoox/Cromwell bootloader. I see you already know certain devices cause a crash, but perhaps this will help you narrow down the problem?

Finally, I seem to be demanding too much from my box, and it's often found thrashing. Is there any possibility of reclaiming more memory back from the video card?
ShALLaX
Site Admin
Posts: 1973
Joined: Sun Aug 10, 2003 9:25 pm
Location: England
Contact:

Post by ShALLaX »

Meh, just setting -X could still make it compile QT.. and all that other crap.

Ive pondered making an rc.conf keyboard editor dealy, I might get round to that sooner or later.

Dont bother with 2.6 just yet, xpad mouse emulation doesnt work nor does Fatx.... but it IS bootable.

We (xbox-linux) are aware of what the bug is... the issue comes with the linux driver being threaded, but cromwell not being (so I'm told).

I believe there is something you can add to your "append" line in the linuxboot.cfg file... videoram=xxxx? maybe?
The original Xbox adaptation of Gentoo
boltronics
Newbie
Posts: 7
Joined: Sat Jan 08, 2005 4:06 am
Location: Hong Kong

Post by boltronics »

Meh, just setting -X could still make it compile QT.. and all that other crap.
I don't think so, so since qt depends on X. Let's take a look:

[ebuild R ] media-video/mplayer-1.0_pre5-r5 -3dfx -3dnow -3dnowex +X +aalib +alsa (-altivec) +arts +bidi -cdparanoia -debug +directfb +divx4linux +doc -dvb +dvd +dvdread -edl +encode +esd +fbcon +ggi +gif +gtk -i8x0 +ipv6 -jack +joystick +jpeg -libcaca +lirc -live -lzo +mad +matroska -matrox +mmx -mmx2 +mpeg -mythtv +nas +network +nls -nvidia +oggvorbis +opengl +oss +png +real -rtc +samba +sdl +sse +svga -tga +theora +truetype -v4l -v4l2 -xanim -xinerama +xmms +xv +xvid -xvmc 0 kB

According to this, if you USE="-X -arts -bidi -esd -ggi -gtk -sdl -truetype -xmms", you shouldn't provde the dependencies that require X.
Dont bother with 2.6 just yet... but it IS bootable.
I don't use the Xpad, nor fatx so these aren't issues for me, but i couldn't get the 2.6 kernel to boot. I wonder if it's a size thing:

Code: Select all

-rw-r--r--  1 root root 1023K Jan 21 11:28 linux-2.4.22
-rw-r--r--  1 root root  1.7M Jan 25 23:07 linux-2.4.28
-rw-r--r--  1 root root  2.8M Jan 27 15:24 linux-2.6.10
Are there any limitations that you're aware of?

Thanks for your replies.
ShALLaX
Site Admin
Posts: 1973
Joined: Sun Aug 10, 2003 9:25 pm
Location: England
Contact:

Post by ShALLaX »

Thats one horribly bloated kernel, but I believe the memory map for Cromwell allows a 3MB kernel.
The original Xbox adaptation of Gentoo
Post Reply