[performance] Speedup plugin retrieval

[ralbertazzi]

terraform plan and terraform apply take a long time to refresh information when working on CloudAMQP projects. By inspecting with -parallelism=1 it looks like the resources that take an unusually long time to be fetched are the plugins.

I can experience the same through the API, where this call takes 3-4 seconds (tried on multiple instances). I suppose it's the same you are using in the provider:

curl -XGET -u :$API_KEY https://api.cloudamqp.com/api/plugins

The first performance optimization would be to simply make this API faster, although it goes outside the scope of this repo. I hope you can still provide this feedback to the people responsible for it.

The second performance optimization involves caching the API call result. I never developed a plugin so I don't know how feasible this is, but since - as far as I see - there is a single API that returns the status of all plugins, you could perform this call just once, cache the result in memory, and use that result to provide the state of subsequent calls. This would greatly increase the refresh time as it wouldn't increase linearly with the number of plugins that are tracked on Terraform.

[dentarg]

Thanks for the feedback around this. We do know the API around plugins are slow and we want to improve them. Any feedback will help us prioritise such work.

Caching (during a run?) in the provider is an interesting idea, but if we can make the API quicker, I think we would prefer that over adding additional complexity to the provider.

[ralbertazzi]

Although not strictly related to this provider, the Plugin API looks currently broken, which means that I'm unable to:

• Check plugins from the UI on multiple instances
• Run any Terraform changes, as the refresh gets stuck

It would be great if you could prioritize an improvement to the Plugin API / resources as it really affects developer experience 🙏

[tbroden84]

We have now deployed a fix to our backend API to address todays plugins retrieval issues. Hopefully this will also solve the speed issue at the same time.

[dentarg]

@ralbertazzi are you still having issues?

[ralbertazzi]

Hi @dentarg , the situation looks much better now, thanks!

I don't know the internals so I don't know how much can still be done to make the API faster. Feel free to close if you believe there isn't much else that can be done.

[dentarg]

Great to hear. The list of plugins is now cached (currently for 5 seconds) after being retrieved (and the cache being invalidated when a plugin is configured). We do want to improve this further but I'm not sure on any timeline. We can keep this open to track it.