Library for easier integration testing of code generators

[derberg]

Reason/Context

It is not so hard to write code generator, it is harder to maintain it ~ Abraham Lincoln

When you write a code generator, you always write it with some use case in mind. You test it manually quite a lot before the initial release. You do it basing on some AsyncAPI file, sometimes crafted for the purpose of the code generator, in most cases it is a specific version of the official streetlight example.

The problem comes with the next PRs that follow. New features that are added, etc. You get contributions about the new protocol that gets supported or support for some feature of AsyncAPI that was not yet supported. You can check if unit tests are added for specific code generation to check if good code will be generated. You can even add snapshot tests to check if generated files look exactly as you wanted them to look like. The problem is that you are never sure if generated code will still work for the previous scenario unless you manually generate an app and test it with a given broker.

Description

As a maintainer of specific template that can generate code, I'd like to have feature in Generator or maybe a separate library that I can easily enable in my repository and enable integration tests for my template:

• Might be that we should just enable this testing feature through https://github.com/asyncapi/generator/blob/master/docs/authoring.md#configuration-file but also might be we need another standalone tool for it (this is most probable)
• As template developer I do not want to write tests like https://github.com/asyncapi/nodejs-template/blob/master/test/integration.test.js. I want it all provided to me by default:
  • by default have test snapshot test that check if generated files match previous snapshot
  • I should be able to specify location of AsyncAPI file that I want to test against
  • I should be able to specify template parameters used in a given test
  • have the option to opt out certain files from test
  • have the option to specify that I expect a specific file to contain specific "text" inside generate file
• As a template developer I want to have a solution in place that will take my AsyncAPI file and generate "application", start a broker if needed, and perform a test operation that will evaluate if generated application is really sending or receiving expected message. Maybe we can integrate https://microcks.io/ ?

For GSoC participants

• you will code with JS or TS
• you will work on a solution that will be used by template maintainers across AsyncAPI org
• you will have a chance to learn in details how to write testing library
• you will have a chance to work with docker, virtualization and testing automation

[sudoshreyansh]

This sounds interesting! I would like to work on this.

[sudoshreyansh]

I am wondering if I can merge this idea with asyncapi/glee#255. AsyncAPI doesn't have any automated testing tool which can perform automated tests based on the specification files, so I can make one and then integrate it with both Glee and Generator. The tool would be able to be used as a standalone tool, similar to the one mentioned in asyncapi/glee#255. It will also expose the necessary functionality needed for integration.

@derberg @fmvilas I need your opinions on this.

[fmvilas]

🤔 @sudoshreyansh I'm not sure we're talking about the same thing here. If I understood correctly, this issue is about generating integration tests for code generators and the one at Glee is about running integration tests with a specific syntax for Glee. As you say in the other issue "While writing tests, developers should only focus on the message being sent to the server and the response provided by the server or the actions the server does."

That said, it may be me who is not understanding properly, so I'm not saying we shouldn't merge the two issues somehow. However, my experience says it's often preferable to start separated and then merge if we think it makes sense. If we start with a single solution for both and we overlooked some details, we'll end up being conditioned on what's possible in the Generator templates or what's possible in the Glee tests. That's just my 2 cents.

[sudoshreyansh]

@fmvilas Thanks! I get you. It would be better if both are separate solutions for now.

[Sihamtahi]

Hello, I want to work on this project. I want to apply my technical knowledge of testing in a real case and have a deep knowledge of this discipline.

[derberg]

@sudoshreyansh this idea was selected for AsyncAPI Mentorship 2022 -> asyncapi/community#376 (comment)

you are the first one that volunteered for this issue + you also contributed to the project a lot by cracking google calendar API for me when I worked on AsyncAPI Meetings automation. It would be great if you decide to be an AsyncAPI-sponsored mentee for this issue 🙏🏼 but I also saw another idea where you volunteered and the idea was selected, so have a look, contact mentor and decide what issue you want to work on during next months

[sudoshreyansh]

@derberg I thought we had to re-pick our ideas for AsyncAPI mentorship after GSoC.
Sorry for the confusion. I would like to go with my application on asyncapi/glee#27

[likitarai1]

Can I work on this issue??

[derberg]

let us wait for @fmvilas decision on asyncapi/glee#27

[derberg]

because @sudoshreyansh works on asyncapi/glee#27 and @Sihamtahi will work on asyncapi/extensions-catalog#78 I will remove (resign as mentor) this issue from the current Mentorship Program in favor of asyncapi-archived-repos/design-system#4 that is complex and requires 2 mentees (and has 2 mentors)

@likitarai1 sorry but the rule was that you need to apply before deadline, so we get people that are really interested in a specific issue. I also do not recall you contributing to the project before, and it is different in the case of folks from asyncapi-archived-repos/design-system#4

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[derberg]

Still valid

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[chinma-yyy]

Hey @derberg , Can we again propose this for the mentorship program this year? I would love to work on this project.

[derberg]

I'm not really sure, this is a big topic, and I proposed 2 others already + have another GSoC topic

Best if someone else from the community could mentor it

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[AyushNautiyalDeveloper]

@derberg could i work on this issue.

[AayushSaini101]

@derberg I am interested in this project Because I recently worked on one of the major features in the generator and we understood the issue that we faced due to testing the changes.

[derberg]

@AayushSaini101 I removed gsoc label. It is not a good issue for mentorship. It is way more complex than I thought when I defined the issue. We learned how complex it is when we tried to do some good integration testing in nodejs template

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[derberg]

we should try to look into it again once we start onboarding first templates in generator

[Gmin2]

Hey @derberg can you suggest me some next steps that i can look into, i am deeply interested in it

[derberg]

@Gmin2 but next steps regarding onboarding templates, or regarding testing lib?

[Gmin2]

Regarding testing libraries 🙂

[Gmin2]

any thoughts @derberg

[derberg]

yeah, we first need to onboard template to generator, and then we can think of more sophisticated end2end testing