Database and In-Memory Support for CSS Generation

[hollyburn]

First off, congrats on writing a useful and powerful tool with a super readable codebase!

Problem

I'd like to use Grimoire as part of a web framework, preferably keeping my config and CSS in a database. It'd be convenient if I could implement a trait (or a few) for this, so I could e.g. deploy my application to a serverless provider or as a single-file container and rebuild my CSS when an admin user modifies the theme.

Solution

Essentially, I'd like a few currently private structs and functions which accept a directory to 1) be pub and 2) instead take perhaps a vector/map filled with reader/writers, or a single reader/writer in the case where they only use the directory to construct one file's path. Having access to say, ConfigJSON would be useful. It'd be nice to ultimately build shared and critical CSS to a string or writer for storage somewhere other than the filesystem, like in a db or hashmap.

Options

Honestly, the language is so well-specified I would be comfortable implementing this in a new crate if it's not something you're interested in for this project. Or with your architectural blessing, I could take a crack at rewriting the relevant parts of this crate to be less tied to the filesystem - without removing any existing fs-based functionality, of course.

[hollyburn]

Wrote a rough-and-tumble little parser for the Spell syntax over at hollyburn/gramaire :)

I'll be refactoring it into a more useful library, adding features as needed for my project - namely identifying Spells in text files, building Scrolls, and writing out CSS. Hope you're well.

[dmtrshat]

Hi @hollyburn,

First of all, thank you so much for your kind words about Grimoire CSS! I’m thrilled that you find the tool useful and the codebase approachable.

Regarding your suggestion, I believe that instead of exposing internal structures and functions, a better approach would be to implement this as an optional module within Grimoire CSS itself. This would maintain the current level of safety while providing the flexibility you’re looking for. By designing it as a feature (e.g., database or alternative storage support), we could keep the core library focused and stable while allowing users to opt-in to this functionality when needed.

I’d love to collaborate and help bring this idea to life! To ensure we address all use cases and design it effectively, I’d like to propose the following steps:

1. Define Requirements: Let’s work together to outline the specific features and requirements. For example:

• What inputs/outputs would you expect the module to handle?
• Which use cases are critical (e.g., specific database integrations, memory storage)?
• How should this integrate with the existing configuration and CSS generation processes?

2. System Design: Once we have the requirements, we can design a robust system to implement these features, ensuring it’s modular and efficient.
3. Implementation: With a clear design, we can integrate this as a feature within Grimoire CSS, avoiding the need for a separate crate and keeping everything in one place.

Let me know your thoughts and if there are any specific ideas or requirements you already have in mind. I’m excited about the possibilities and am more than happy to actively develop this alongside you to ensure it aligns with your needs.

Looking forward to hearing from you!

[hollyburn]

Hi!

I briefly mentioned it in my first post, but I think the way to go if you're willing to change large parts of the code would be structuring Grimoire to use traits for input and output. This way, the existing filesystem code could impl e.g. GrimoireIO (or maybe separate GrimoireInput and GrimoireOutput traits); the CLI would use this default fs implementation, or have a flag to select between a few included backends. Such a trait could look something like:

pub trait GrimoireIO<'a> {
    /// an iterator over critical-path Spells specified as strings
    fn critical_spells() -> impl Iterator<Item = &'a str>;
    /// an iterator over readers representing critical-path input css files to be fed to the css minifier
    fn critical_css() -> impl Iterator<Item = impl Read>;

    /// an iterator over shared Spells specified as strings
    fn shared_spells() -> impl Iterator<Item = &'a str>;
    /// an iterator over readers representing shared input css files to be fed to the css minifier
    fn shared_css() -> impl Iterator<Item = impl Read>;

    /// an iterator over readers which might contain spells, and a key so they can be associated to an output
    fn input_files() -> impl Iterator<Item = (impl Hash, impl Read)>;

    // and so on and so forth... :)
}

This is hard to specify without a full config.json which uses all the features of Grimoire. A config with couple of example projects would be very helpful to include in the repo for testing. This would make it much easier to maintain all the existing functionality when specifying an API.

[dmtrshat]

Thank you! I really appreciate the thought and effort you’ve put into this.

This is an important architectural decision, and I’d like to take some time to fully evaluate the implications and possibilities. I’ll get back to you by next week with my thoughts and a more concrete plan.

[dmtrshat]

Hi @hollyburn,

I’ll be working on your suggested feature as part of the v1.3.0 Liminal: Beyond the Threshold release. Development is planned to take place from December 23, 2024, to January 6, 2025, after the current v1.2.0 Emberveil: Transformative Styling release is completed.

This release will also include other improvements. Once I’ve outlined the steps, architecture, and tasks for this feature and the release, I’ll share them so you can check it out if you’re interested.

Let me know if you'd like to contribute.

[hollyburn]

That sounds great! I'd be happy to help in any way I can.

[dmtrshat]

Here is the feature overview:

We aim to make Grimoire CSS more flexible by allowing two distinct usage modes:

1. File-based mode (existing Config structure)

   • Reads input paths from the filesystem; writes compiled CSS to files.

2. In-memory mode (a new ConfigInMemory structure)

   • Accepts raw CSS data or user-defined strings directly in memory; returns compiled CSS as data structures instead of forcing file output.

This enhancement means Grimoire can be easily integrated into serverless environments or web frameworks where storing both configuration and final CSS in a database (or a remote store) is preferred.

---

Key Milestones / High-Level Steps

1. Define a New ConfigInMemory .

2. Add Dedicated Functions/Methods for In-Memory Building (these methods parse or transform in-memory data and produce a compiled CSS result without reading/writing the disk).

3. Adapt Internal Parsing/Compilation.

4. Retain File-Based Logic (ensure the existing file-based flow remains untouched).

5. Document and Test.

---

Why This Approach Is Preferable over a Trait-Based IO Layer

1. Simplicity

   • A trait-based IO approach might demand rewriting many methods to support arbitrary streaming or readers/writers. Given that CSS parser expects a full string, streaming offers little advantage.
   • Maintaining a large trait with many streaming or iterator-based methods could be overkill if most real-world scenarios only need “read full content / produce full content.”

2. Clear Separation

   • By having Config for the file-based approach and ConfigInMemory for the in-memory approach, it’s immediately obvious which mode you’re using. There’s no confusion with half-populated fields or conditional checks.

3. No Changes in the Existing Flow

   • We don’t break existing file-based workflows or rename existing APIs. Current users can keep doing what they do, unaffected.
   • We add a brand-new, simpler path for those who want direct memory usage.

4. Practical Coverage

   • Nearly all feature requests revolve around “I want to store data in a DB or memory, then feed it to Grimoire, and get the result as a string.” That’s exactly solved by an in-memory config model, without the overhead of building a generic trait-based streaming system.

Hence, this approach balances ease of implementation, and comprehensive coverage for typical scenarios - especially given the nature of CSS-parsers working on complete CSS strings.

---

The board with tasks is ready as well. Feel free to take a look! 🙃