Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve testability of source generators providing Verify into project #2691

Open
KondzioSSJ4 opened this issue Sep 9, 2024 · 8 comments · May be fixed by #2708
Open

Improve testability of source generators providing Verify into project #2691

KondzioSSJ4 opened this issue Sep 9, 2024 · 8 comments · May be fixed by #2708
Labels

Comments

@KondzioSSJ4
Copy link

Description

Hi

I tried to make some improvements of the code by making a generator and providing a example how it can looks
but...
I have a problem with that tests because:

  • almost all the time you need to exact know what is the output
  • updating the output of source generator can be painful
  • comparing of "before" and "after" is not easy

Can I provide a pull request with potential way of how improve tests and how we can verify the output?

Also... I'm not sure about license of the external library, if that is acceptable by your team or not
Link to that library: https://github.com/VerifyTests/Verify.SourceGenerators

@KondzioSSJ4
Copy link
Author

I tried to make separated branch and put there a sample how it can look but I get:

unable to access 'https://github.com/Azure/azure-functions-dotnet-worker.git/': The requested URL returned error: 403

so...
There is the patch file (ZIPped because github not supported other files -_- )

patch.zip

@kshyju
Copy link
Member

kshyju commented Sep 9, 2024

@KondzioSSJ4 We welcome contributions to improve the product. Would you mind elaborating the problem you were trying/suggesting to solve? Apologies, I did not understand it totally.

@KondzioSSJ4
Copy link
Author

@kshyju
Maybe i'm just bad with your way of testing
but my issue is to provide easy way to making a source generators
to open a way to easier making generators for AoT

For example...
After adding Verify you need to make a small test like:

string inputCode = """
using System;
using System.Collections.Generic;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;

namespace FunctionApp
{
    public static class HttpTriggerSimple
    {
        [Function("HttpTrigger")]
        public static void HttpTrigger([HttpTrigger(AuthorizationLevel.Admin, "get", "post", Route = "/api2")] HttpRequestData req)
        {
            throw new NotImplementedException();
        }
    }
}
""";

await new FunctionMetadataProviderGenerator()
    .RunAndVerify(
        inputCode,
        _referencedExtensionAssemblies,
        languageVersion: languageVersion);

And then run a test
every time when you run test it try to check what was "verified" and what was returned
and it's comparing all generated files and shown in e.g. Visual Studio Code (or other tool that support diff files) the differences if there was any, if was a difference then fail a test
(then you need to make verified file match to expectation, like "coping left to right")

Also many diff files tools (like VS Code) shown syntax, so it's easier to check what's wrong with the output

but main goal of this is allowing to make easier generators
because I don't need to see the output at the beginning
but I can use different emit methods to display what was the result of each step
e.g. first check if the functions are correctly loaded from the syntax and if the input of emiter is what I expected
then we can do next step like providing expected emiter and verify that output

Benefits:

  • easier to make generators in steps
  • easier way to accept changes in generator output
  • syntax coloring in generated output

I know that there are some strange changes like:

  • updated version of Roslyn package to 4.11 to allow see incremental generators
  • instaled packaged "Microsoft.CodeAnalysis.Features" Version="4.11.0" and "Microsoft.CodeAnalysis.VisualBasic.Workspaces" Version="4.11.0" and that's because updating Roslyn enforce to have all packages in the same version, but one of other package (SmartAnalyzers.RoslynTestKit) enforce to have VisualBasic and Features DDLs, and without enforcing that package version provides to a error of inconclusive version

@SimonCropp
Copy link
Contributor

@KondzioSSJ4

re

unable to access 'https://github.com/Azure/azure-functions-dotnet-worker.git/': The requested URL returned error: 403

did you fork the repository before trying to push the branch?

@SimonCropp
Copy link
Contributor

@KondzioSSJ4

what i dont understand about this issue and the patch is what you are trying to achieve?

if you are proposing the azure-functions-dotnet-worker team use snapshot testing, i recommend starting with a smaller and simpler scenario. so you can explain the value of snapshot testing without the significant noise of a complex scenario

if you want to use snapshot testing when building your azure functions, you should be able to do that without any changes in this repository

if you have hit problems using snapshot testing to test your azure functions, then perhaps share a simple example of that problem

@KondzioSSJ4
Copy link
Author

KondzioSSJ4 commented Sep 10, 2024

@SimonCropp

what i dont understand about this issue and the patch is what you are trying to achieve?

Mostly... Easier way to make tests in project to validate source generated code via snapshot tests

Also... side effect would be learn how I can contribute in this project


if you are proposing the azure-functions-dotnet-worker team use snapshot testing, i recommend starting with a smaller and simpler scenario. so you can explain the value of snapshot testing without the significant noise of a complex scenario

I already made in the patch file that I provided before
but there is branch with that changes
main...KondzioSSJ4:azure-functions-dotnet-worker:issues/2691-adding-snapshot-source-generator-testing


if you want to use snapshot testing when building your azure functions, you should be able to do that without any changes in this repository

Nope, I would like to improve and make some source generators for AoT but current way of providing test is painfull and any change of existing source generator output required much more developer attention that it should be

@SimonCropp
Copy link
Contributor

Also... side effect would be learn how I can contribute in this project

then i suggest you try much smaller and incremental changes that gradually improve this project.

for example, i notice your change requires certain nugets to be updated. perhaps start with that as your first contribution

@KondzioSSJ4
Copy link
Author

@SimonCropp
Ok, but even small changers required to deal with all that tests
for example...
changing source generator of line:

var {{functionBindingsListVarName}} = new List<string>();

(In method AddFunctionMetadataInfo inside of FunctionMetadataProviderGenerator.Emitter)
To change to have a List<string>(<capacity>) it's required to go to almost every test and manually change line matching to that number
what sounds painful, because you exactly know where to change and comparing to existing value
also... comparing even in github looks bad because all this code is a string from github perspective
and I get it... you used to that way, but probably for normal human begins it's harder
and making a tests easier to write and maintaining sounds as good goal to achieve

@KondzioSSJ4 KondzioSSJ4 linked a pull request Sep 12, 2024 that will close this issue
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants