Grapher 2.0 to support Meteor 3.0

[donstephan]

Starting to think about how to support Meteor 3.0 and the changes that are going to need to happen in grapher. Some thoughts:

• I think we should skip intermediary support and jump right to the removal of fibers. You either use grapher with fibers or you don't.
• We need to support async filters and reducers. This should be easier since there will be a top level await.

There are currently 4 other community/third party packages used in grapher. It's worth exploring which ones of those will cause issues:

• matb33:collection-hooks - Keep an eye on it. Definitely needs updates. This is used on linker removal. If this isn't updated we can just recreate the after.remove hook and remove this package.
• reywood:publish-composite - Don't think we should have any problems with this one. I looked through it and didn't see anything that shouldn't be compatible with the removal of fibers.
• dburles:mongo-collection-instances - No issues.
• herteby:denormalize - Lots of collection hooks in this one. Also going to need some updates in it to make it async.

I'm waiting for the 2.8 release to get started on this but let me know if you have any other thoughts.

[StorytellerCZ]

We can update them all officially, I have the access right via MCP. Once Meteor 2.8 is released the work is going to start to migrate all of them to the new async functions. Internally immediately and for the external API I think it is going to make a switch with Meteor 3.0 as well. In the meantime there are going to be beta releases with the *Async used to get things ready.

[BastienRodz]

Hello ! My company and I are using Grapher a lot on our Meteor projects and we wanted to know if there is any update to this situation ?
We are preparing to get our old Meteor projects ready for the 3.0 and we don't know how to deal with Grapher in that mountain of work.
Thanks a lot for your reply and your amazing work 😄

[paulincai]

@BastienRodz do you use it with publications or methods only?

[BastienRodz]

@BastienRodz do you use it with publications or methods only?

We use both.
Used to use a lot of publications, now we rely more on methods.

[paulincai]

Hey Bastien. Grapher is a community package which indeed means that it is maintained by the community. I am trying to understand the position of your company and yours so let me see if I understand this correctly:
Your company is waiting for someone in the community to update this package for free so that they can make money with it?

Let me please answer to your initial question. The update is that your company may volunteer to push the update for this package and once the update is complete you will know exactly how to deal with Grapher.

Open source doesn't mean take for free. It also means give.
You can also offer a bounty for the community. Given the nature of Grapher, I see at least 8-10 days of work (if it includes completely TS-ing the package) so something in the area of 1500 Euros is what your company might want to think of.

Alternatively you can wait for someone to need it more than your company and perhaps they will be kind to push it to the free community packages repo.

[BastienRodz]

Hey @paulincai, thanks for your quick response!

I totally get your point about the open-source community and the importance of contributing. Actually, my company is a big supporter of Meteor, and we've actively contributed to various packages in the past. Our question was more about understanding the current status and roadmap for Grapher in the context of Meteor 3.0, so we can plan accordingly.

We're definitely open to contributing to the update of Grapher, especially since we rely on it heavily. If there’s an ongoing effort that we can join or support, that would be great to know. Alternatively, if it's something that hasn't been started yet, we're open to discussing how we can take the lead or support this initiative.

Again, sorry for any confusion my initial message might have caused. We're all in this together, and we're more than willing to pitch in and help out!

[StorytellerCZ]

I'll be happy to merge any relevant PRs, but I'm not using Grapher myself at this point and I'm too busy to work on this beyond merging PRs and publishing new releases.

---

https://github.com/sponsors/storytellerCZ/

[bhunjadi]

On the project I'm working on, we're preparing for Meteor 3.0 as well.
Since I contributed to grapher in the past, I know my way around the project and I think I can check in the next two weeks.

I would advise against doing TS + Meteor 3.0 both at the same time. Either we do TS migration first or we go with Meteor 3.0 first and then TS.

[paulincai]

@BastienRodz I think there was no confusion :). It just happens that there is more and more of "hey guys when do you update this or that package". One thing we forget is that the community is all of us and a normal practice for many tech projects in the open source is to have your company committed with a monthly sponsorship behind the public project. (photo for reference).

I really encourage you to have one of the main (or more) knowledgeable Meteor contributors on your permanent sponsorship list and avail of their time to do updates in your own timeline or to help you with migrations or anything else.

If I see your company decently sponsoring any of the contributors such as @StorytellerCZ (above) I promise you that I will contribute myself at least 16 hours worth of work on Grapher for Meteor 3. The earlier your company can do it, the earlier you will be able to start testing your Meteor 3 project with the new Grapher.
Otherwise, as I mentioned in my previous message, you can start PR-ing on the project or develop it locally or wait for someone to jump on the task. One thing you should not worry about is that Grapher will be deprecated, unmaintained etc. I can assure you that that will not be the case.

I think the situation si very clear now:

1. There is an alpha for Meteor 3 to work with.
2. There is a list of priority packages (MCP) that need to be updated.
3. A document and video on how to migrate packages exists.
4. All packages are public and can receive PRs from anyone with a Github account.

[bhunjadi]

I started working on Grapher compatible with Meteor 3.0 in #484

[paulincai]

I started working on Grapher compatible with Meteor 3.0 in #484

Hi @bhunjadi, I keep here an async version of Grapher. I did this update a couple of weeks back and ran my queries with this version within a Meteor 3 alpha project. The code lint is Standard (which you might not like) but if you search for async await in it, you will find the relevant changes required for the use of Methods. I do not use reactivity with Grapher so whatever involves reactivity was not touched: https://github.com/paulincai/grapher/tree/master

[bhunjadi]

Thanks @paulincai !

[vparpoil]

Hi Everyone,
I wanted to add some thoughts to the on-going discussion and work about making grapher compatible with Meteor 3.

Reading this successfull testimonial of a migration to its succesor Nova, I DM'd @Twisterking and he confirmed it was easy and working flawlessly so we have decided to go this route on our way migrating to Meteor 3.

I think the @bluelibs/nova version of the fetching layer is an evolved version of the one included in cultofcoders:grapher. Dropping support to meteor and subscriptions must have simplified parts of the code, and I'm sure @theodorDiaconu and his team used their extensive experience to create a better version of Grapher. I also spotted some new functionalities in the documentation that fix some issues we had using grapher in the past (like one-to-one relations and filtered links), so I'm quite excited to use it for a new project.

I am a bit concerned by the relatively low activity of the bluelibs repo, but I don't see many bugs on the issues list, and anyhow, I prefer to rely on a lib that is actively used by his creator than the previous lib that is not anymore in maintenance mode.

I'm bringing this comment here to share how we integrated @bluelibs/nova to our codebase to have minimal changes, and to question the necessity for MCP to continue supporting grapher (I'm sorry for being the one who brought this matter initially) : MCP and @StorytellerCZ have a lot to do already, and I think nova is a robust alternative to grapher.
The drawback to stop support will be that existing apps will need some code rewrite to transition to Meteor 3 (which is obviously already needed), and I think most of the apps in maintenance mode will never be upgraded, except the one under active development of new features.
Maybe one route would be to create another atmosphere package bringing Meteor integration with Nova
What are you thoughts on this matter?

Our progress on integrating @bluelibs/nova

Warning

This is on progress work, we haven't tested yet extensively

We don't use grapher on the client anymore (we did in the past), and I think this is a pre-requisite to transition to nova :

• we define queries and links server-side, we don't use exposure
• we use grapher with named queries that we define server-side and that we call from meteor validated methods or other server-side code

We created two utilities functions to wrap the addLinks function and the addReducers functions and we use them on our own extended version of Mongo.Collection. We needed to store the rawCollection() in a local variable to have only one reference (this comment of Theodor led us on this path)

Here is the code of our utilities

import { Mongo } from "meteor/mongo";
import { addLinks, addReducers } from "@bluelibs/nova";

export const MongoCollection = class extends Mongo.Collection {
    constructor(props) {
        super(props);

        // for @bluelibs/nova, we need to have only one instance of raw collection and use it everywhere
        if (Meteor.isServer) {
            // rawCollection is only available on server
            this.raw = this.rawCollection();
            this.addReducers = addReducersFunction;
            this.addLinks = addLinksFunction;
        }
    }
};

/**
 * A wrapper for Nova addReducers function to use Meteor collections directly
 * @param reducers
 */
export const addReducersFunction = function (reducers) {
    // test if reducers are Async to help migrating previously written code
    let reducerKeys = Object.keys(reducers);
    reducerKeys.forEach((reducerKey) => {
        let reducer = reducers[reducerKey];
        if (!(reducer.reduce({}) instanceof Promise)) {
            console.warn(`The reduce function of the reducer ${reducerKey} must be an async function`, reducer.reduce.constructor.name);
        }
    });
    return addReducers(this.raw, reducers);
};

/**
 * A wrapper for Nova addLinks function to use Meteor collections directly
 * @param links {Object} - links object
 */
export const addLinksFunction = function (links) {
    // the collection key must be updated
    let newLinks = {};
    let linkKeys = Object.keys(links);
    linkKeys.forEach((linkKey) => {
        let link = links[linkKey];
        newLinks[linkKey] = {
            collection: () => link.collection.raw,
            field: link.field,
            foreignField: link.foreignField,
            unique: link.unique,
            many: link.many,
            inversedBy: link.inversedBy,
            index: link.index,
            filters: link.filters,
        };
    });
    return addLinks(this.raw, newLinks);
};

Those two utilities make our code exactly the same for defining links and reducers, except for the exceptions of Meteor.roles (from alaning:roles), Meteor.users, and FilesCollections created using ostrio:files. Indeed those collections do not use our own extended Mongo.Collection version. For those collections, I had to bind addLinksFunction and addReducersFunction to the collections, but I'm sure this can be simplified. This brings immediate support for async reducers.

The last step was to integrate query to the queries, and there I haven't found yet how to do it without rewriting the code, and I didn't wanted to create a translator for the bodies of the queries since the signature of filters, options etc changed.

Here is an example of a working query with Meteor users and roles and the binding mentioned earlier

addLinksFunction.bind(Meteor.roleAssignment)({
    user: {
        type: "one",
        collection: Meteor.users, 
        field: "user._id", 
    },
});

addLinksFunction.bind(Meteor.users)({
    roles: {
        type: "one", 
        collection: Meteor.roleAssignment,
        inversedBy: "user",
    },
});

addReducersFunction.bind(Meteor.users)({
    firstName: {
        body: {
            profile: {
                firstName: 1,
            },
        },
        async reduce(doc) {
            return doc.profile?.firstName;
        },
    },
    lastName: {
        body: {
            profile: {
                lastName: 1,
            },
        },
        async reduce(doc) {
            return doc.profile?.lastName;
        },
    },
    email: {
        body: {
            emails: 1,
        },
        async reduce(doc) {
            return doc?.emails && doc.emails[0].address;
        },
    },
});


import { Meteor } from "meteor/meteor";
import SimpleSchema from "meteor/aldeed:simple-schema";
import { query } from "@bluelibs/nova";

Meteor.users.grapherQueries = {};

Meteor.users.grapherQueries.getUsers = (params) => {
    // validate params
    new SimpleSchema({}).validate(params);

    return query(Meteor.users.raw, {
        $: {
            $filter({ filters, options, params }) {
                // filters.field = params.field
            },
            // $options: {
            //     sort: {field: 1}
            // },
            //$paginate: false,
        },

        // Direct fields
        createdAt: 1,
        firstName: 1,
        lastName: 1,
        emails: 1,
        date: 1,

        // links
        roles: {
            role: 1,
            scope: 1,
            inheritedRoles: 1,
        },

        // Reducers
    });
};

[pmogollons]

@vparpoil did you have to install mongodb package or you found a way to fix the mongodb module not resolved error?

[vparpoil]

@pmogollons Yes, I had to install mongodb, and I have an unfriendly warning when starting the app :

Unable to resolve some modules
  "bson-ext" in ./node_modules/mongodb/lib/bson.js (web.browser.legacy)
  "mongodb-client-encryption" in ./node_modules/mongodb/lib/utils.js (web.browser.legacy)
  "kerberos" in ./node_modules/mongodb/lib/deps.js (web.browser.legacy)
  "@mongodb-js/zstd" in ./node_modules/mongodb/lib/deps.js (web.browser.legacy)
  "http2" in  ./node_modules/@smithy/node-http-handler/dist-cjs/index.js (web.browser.legacy)
  "snappy" in ./node_modules/mongodb/lib/deps.js (web.browser.legacy)
  "snappy/package.json" in ./node_modules/mongodb/lib/deps.js (web.browser.legacy)
                                              
If you notice problems related to these missing modules, consider running:
                                              
  meteor npm install --save bson-ext mongodb-client-encryption kerberos @mongodb-js/zstd meteor-node-stubs snappy

but still it works

I only have one missing feature : links to Meteor.users collection don't work yet

[pmogollons]

Interesting, what version of mongodb do you have installed? In my case I get the same warning but the app doesnt start and then I get the error

=> Errors prevented startup:                  
   
   While building for web.browser:
   node_modules/bson/lib/bson.mjs:234:28: Unexpected reserved word 'await'. (234:28)
   
=> Your application has errors. Waiting for file change.

Edit: Version 4.17.0 is the version that works

[pmogollons]

@vparpoil how are you setting the Meteor.users.raw?

[pmogollons]

I was able to create a basic "compat layer" package to bring nova to grapher apps with less changes. It uses the same queries, namedQueries (with resolvers, exposures and cache). Ill try to have a repo with this later this week.

[theodorDiaconu]

Hello gentlemen,

BlueLibs is currently in stability zone. Any bug that appears will be squashed fast. I personally cannot sleep knowing a bug exists in BlueLibs. We use it extensively in live projects, even projects that make 100k+ revenue a month, there are big financial interests in keeping BlueLibs alive, stable and secure.

Nova is part of BlueLibs and we treat it with same care.
Nova is a downgrade feature-wise from Grapher, Grapher has been designed for Meteor, Nova has been designed for MongoDB with a GraphQL-oriented touch.

If any problems or you need me, tag me pls or just send me an email at [email protected] I will respond to everyone needing my help :)

[vparpoil]

@vparpoil how are you setting the Meteor.users.raw?

Well, Meteor.users.raw = Meteor.users.rawCollection()
But my problem seems to have gone away....

[pmogollons]

well here is a version of what I was talking about before.

Grapher + nova

Have in mind that:

• Is not compatible with pub/sub
• It has no meta links
• No linker engine
• No $postFilters and $postOptions
• No denormalization
• No global or collection expose
• No graphQL bridge

Documentation can be improved a lot, but its mostly compatible with the old version of grapher except for the above.

To try it out just git clone it to your packages, add meteor/pmogollons:nova to your packages file and have in mind the following:

• Removed fetch, fetchSync, fetchOne and fetchOneSync, now use fetchAsync and fetchOneAsync
• You must instance your collection with the custom MongoCollection from this package
• No support for $filters, $filter and /$options in reducers and links body, use $: { filters: {}, options: {} } instead
• All reducers reduce function should be async
• foreignIdentityField is now foreignField
• For users links and reducers you need to use the export functions

You must instance your collection with the custom MongoCollection from this package

import { MongoCollection } from "meteor/pmogollons:nova";


const Suppliers = new MongoCollection("suppliers");

export default Suppliers;

For users links and reducers you need to use the export functions

import { Meteor } from "meteor/meteor";


if (Meteor.isServer) {
  Meteor.users.raw = Meteor.users.rawCollection();
}

addLinksFunction.bind(Meteor.users)({
    roles: {
        type: "one", 
        collection: Meteor.roleAssignment,
        inversedBy: "user",
    },
});

addReducersFunction.bind(Meteor.users)({
    firstName: {
        body: {
            profile: {
                firstName: 1,
            },
        },
        async reduce(doc) {
            return doc.profile?.firstName;
        },
    },
});

Let me know if it works for you or if you find any issues.

[bhunjadi]

Just to post an update here, migration to Meteor 3.0 is mostly done and we're probably ready to release a beta version. I updated denormalize package to Meteor 3.0 and this was the last obstacle that I had.

All tests are passing now on Meteor 3.0, but I expect some additional fixes will be required when people start testing it out.

PS
If you can go @bluelibs/nova route, I would definitely recommend it. For my use case, the app is heavily using Grapher and pub/sub & denormalize so this is not at option while migrating to Meteor 3.0.