modularize features

[timkenhan]

Makes features (e.g. xfsprogs, e2fsprogs, plymouth, luks, etc) more self-contained thru the new GK_FEATURES variable

Some changes on the ebuild script will be necessary to accommodate this change.

To ensure working initramfs will always be generated, even without GK_FEATURES set, some basic functionalities such as busybox, firmware, and kernel modules will always be included.

I am aware this work is far from finish, and thus am open for input!

Some (minor) blockers:

• I'm having issues separating NFS & netboot features since it's quite tightly coded with busybox stuff within the script
• despite dmraid, lvm, luks, and multipath resides on their own, append_data 'devicemanager' "${DMRAID}" "${LVM}" "${LUKS}" "${MULTIPATH}" is on the main file.

[timkenhan]

@reagentoo @FlyingWaffleDev feel free to toy around with this!

[reagentoo]

Hi. Thank you for your effort!
What are the purposes of these changes?

1. Reducing the initrd image size?
2. Accelerate initrd image loading?
3. Accelerating initrd image generation?
4. Make code easier to understand?
5. Improved code stability?
6. Improve security?

[timkenhan]

I say making the code easier to understand, especially for adding new features. For example, in the recent implementation of Plymouth feature, @FlyingWaffleDev would need to figure out where to put what within multiple files of which each file can have hundreds of lines. With this change, only one file would need to be added.

The user (or the packager) can also choose which features to have in the genkernel executables. I intend to pair this with USE flag to give Gentoo users ability to pick and choose.

[reagentoo]

If you are saying only about code easier to understand, why did you make the code generation?

[timkenhan]

It's not only about making the code easier to understand. It's also about giving users ability to pick and choose features on their executables.

[timkenhan]

As for the readability, I am making the source files less monolithic, thus letting developers able to do their work without sifting thru hundreds (or thousands) lines of code.

[reagentoo]

It's also about giving users ability to pick and choose features on their executables.

Does this "choising ability for users" improves at least 1 point out of those six? #52 (comment)

As for the readability, I am making the source files less monolithic, thus letting developers able to do their work without sifting thru hundreds (or thousands) lines of code.

I don't agree. Any genkernel's shell file within 100kb.

[timkenhan]

i guess it does, but not directly (since if an option is included, it can still be turned off).

100kb is massive for a textfile. Some of the scripts would reach 1k loc.

it is also more prone to conflict we have fewer big files, as compared with many small files.

[thesamesam]

This should at least let us make some of the large distfiles in the ebuild optional too as they'll be real build system options rather than useless USE flags just toggling a download.

[reagentoo]

This should at least let us make some of the large distfiles in the ebuild optional too as they'll be real build system options rather than useless USE flags just toggling a download.

To implement this, it is not necessary to do code generation and divide normal-sized sources into a bunch of small files.

[thesamesam]

This should at least let us make some of the large distfiles in the ebuild optional too as they'll be real build system options rather than useless USE flags just toggling a download.

To implement this, it is not necessary to do code generation and divide normal-sized sources into a bunch of small files.

It is to have USE flags which aren't there for the sake of it. But besides, I agree with the general splitting up of things here. The files are huge and dividing things up (forcing us to think about their interface) is not a bad thing.

[reagentoo]

forcing us to think about their interface

There is no need to do such code generation for the interface (using awk 🤦‍♂️).
It is enough to use smth like:

source ${module}
interface_func_${module}
interface_var+=( "item" )

[thesamesam]

forcing us to think about their interface

There is no need to do such code generation for the interface (using awk 🤦‍♂️). It is enough to use smth like:

I suggest making more constructive, polite comments.

source ${module}
interface_func_${module}

Sure, propose that then.

[timkenhan]

@reagentoo: If you read the awk script, you'd see the reason I did the way I did was so that we can generate any section with the same piece of code. There's also no need to install an extra package since awk is required for most system packages anyway. Can I ask what you mean by "no need for the code generation"? What is wrong with awk?

[reagentoo]

Can I ask what you mean by "no need for the code generation"?

Are there really no other approaches to creating an interface and modules?

What is wrong with awk?

Can you give an examples of a good practice for using awk in this way? Worse than awk - only perl.

[timkenhan]

Are there really no other approaches to creating an interface and modules?

I guess we can have the master code calling the module code, but that would take further approach. We need to start with some degree of separation to move forward. I'd be open to suggestion on the implementation detail.

Can you give an examples of a good practice for using awk in this way? Worse than awk - only perl.

Please answer the question, what is wrong with awk? I simply need a tool for processing text, and awk simply fulfils my need with its specialized language for text processing. Once again, I am open for suggestion.

[reagentoo]

Please answer the question, what is wrong with awk?

Everything is fine with awk, just like with perl.

Once again, I am open for suggestion.

I suggest to not use text processing.

I'd be open to suggestion on the implementation detail.

I'm sure that you can get by with the built-in sh/bash capabilities for organizing modules and interfaces (w/o text generation).

[thesamesam]

Does it really need gawk, or just any posix awk? Does mawk work?

[timkenhan]

I guess it would work with any awk. I've tried it with mawk and it works. Should I update it accordingly?

[thesamesam]

yes please

[timkenhan]

done!

[timkenhan]

@thesamesam lemme know if you have other feedbacks :)

[robbat2]

This won't trigger properly on updates.
If I edit something in genkernel.conf and run make again, out/genkernel.conf will not be regenerated.

It's also unclear why it depends on out/build-config when it does NOT consume that file at all.

out/genkernel.conf: genkernel.conf out/temp/genkernel_conf
    cat $< | sed -e '/#BEGIN FEATURES genkernel_conf/ r out/temp/genkernel_conf' >$@

The same issue exists for many of your other targets.

I suggest creating a helper script that supports every transformation you've coded into the Makefile, and then simplifying the makefile:

out/genkernel.conf: genkernel.conf out/build-config $(INTERMEDIATE_DEPS) helper.sh
  ./helper.sh < $< > $@

and then you can see how it's easy to convert that rule further (this is a little harder without changing the layout to have the source files in a distinct directory).

out/%.conf: %.conf out/build-config $(INTERMEDIATE_DEPS) helper.sh
  ./helper.sh < $< > $@

[timkenhan]

I created the build-config despite not being directly consumed because i was having problem getting the proper dependencies to register for preventing rebuilds. I thought about having separate dependencies for the temp files, but they are all processed at once within the awk script.

[robbat2]

missing deps here

[timkenhan]

can you elaborate on this? i've put the deps on top of file

[robbat2]

Overall I like the concept you're proposing here. I'm worried about the long-term maintainability of the split implementation. One immediate loss is that we can no longer quickly do shellcheck to detect bugs - because it can only check the output files.

Make also isn't suited to this type of text munging. M4 might work, but I wouldn't wish it on anybody.

[timkenhan]

@robbat2 Thank you for your massive list of feedback! This should deepen my understandin of Makefiles. I'll address them as soon as I can.

What we have here is basically the proof of concept to demonstrate that this can work. To be honest, I find the implementation itself to have a long way to go. The feature file format is far from perfect, but it is one I could come up with. Let me know if you have better ideas!

[timkenhan]

hi @robbat2, I've made some restructuring changes on the features. I hope that this would suffice to address your concerns above.

[thesamesam]

Robin's travelling until the end of the month btw (https://dev.gentoo.org/devaway/)

[timkenhan]

Thanks for the heads-up, @thesamesam!

[timkenhan]

@thesamesam @robbat2 any word on how we can proceed?