[Module Proposal]: Azure Data Explorer Clusters [Kusto/clusters] #287

vlahane · 2023-03-23T05:56:35Z

Have you checked this module does not already exist in the registry?

Yes

Why is the module needed?

We require a base module for managing KustoClusters and one is not currently available in the public registry. It will follow the concept of existing modules in bicep-registry-modules repo.

We will be needing this bicep module to provision Kusto Clusters.

Module path

storage/data-explorer

Describe the module

This module will allow for provisioning and management of KustoClusters (Azure Data Explorer Clusters), in line with a similar experience as using Azure Portal/CLI. Including support for Private endpoint, Diagnostic Settings,etc

module name we can also make as data-analytics/data-explorer-clusters let's me know if any suggestions for module path.

The text was updated successfully, but these errors were encountered:

dciborow · 2023-04-05T08:17:10Z

for module path, how about 'storage/data-explorer'?

Here is the output from my prototype GPT model based on the issue.

main.bicep

@description('Deployment Location')
param location string

@description('Name of the Kusto Cluster. Must be unique within Azure.')
param name string = 'kusto${uniqueString(resourceGroup().id, subscription().id)}'

@description('The SKU of the Kusto Cluster.')
param sku string = 'Dev(No SLA)_Standard_D11_v2'

@description('The number of nodes in the Kusto Cluster.')
param numberOfNodes int = 1

@description('The version of the Kusto Cluster.')
param version string = 'latest'

@description('The tags of the Kusto Cluster.')
param tags object = {}

resource kustoCluster 'Microsoft.Kusto/clusters@2020-02-15' = {
  name: name
  location: location
  sku: {
    name: sku
  }
  properties: {
    numberOfNodes: numberOfNodes
    version: version
    tags: tags
  }
}

@description('The ID of the created or existing Kusto Cluster. Use this ID to reference the Kusto Cluster in other Azure resource deployments.')
output id string = kustoCluster.id

Summary

This Bicep module creates a Kusto Cluster with the specified number of nodes and version.

Description

Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more. Azure Data Explorer clusters are the fundamental resource in Azure Data Explorer. A cluster is a collection of compute resources that host databases and tables.

This Bicep module allows users to create or use existing Kusto Clusters with options to control the number of nodes and version. The output of the module is the ID of the created or existing Kusto Cluster, which can be used in other Azure resource deployments.

Examples

Example 1

This example creates a new Kusto Cluster with a unique name in the East US region using the default Kusto Cluster configuration settings. The module output is the ID of the created Kusto Cluster, which can be used in other Azure resource deployments.

module kustoCluster 'br/public:data-analytics/kusto-clusters:0.0.1' = {
  name: 'mykustocluster'
  params: {
    location: 'eastus'
  }
}

output kustoClusterID string = kustoCluster.outputs.id

Example 2

This example creates a new Kusto Cluster with the name "mykustocluster" in the "myresourcegroup" resource group located in the East US region. The Kusto Cluster is configured to use a specific version and number of nodes. The module output is the ID of the created or existing Kusto Cluster, which can be used in other Azure resource deployments.

param location string = 'eastus'
param name string = 'mykustocluster'
param resourceGroupName string = 'myresourcegroup'
param version string = 'latest'
param numberOfNodes int = 1

module kustoCluster 'br/public:data-analytics/kusto-clusters:0.0.1' = {
  name: 'mykustocluster'
  params: {
    location: location
    name: name
    resourceGroupName: resourceGroupName
    version: version
    numberOfNodes: numberOfNodes
  }
}

output kustoClusterID string = kustoCluster.outputs.id

test.bicep

/*
Write deployment tests in this file. Any module that references the main
module file is a deployment test. Make sure at least one test is added.
*/

param location string = resourceGroup().location

//Prerequisites
// module prereq 'prereq.test.bicep' = {
//   name: 'test-prereqs'
//   params: {
//     location: location
//   }
// }

//Test 0. 
module test0 '../main.bicep' = {
  name: 'test0'
  params: {
    location: location
  }
}

vlahane · 2023-04-05T11:01:21Z

for module path, how about 'storage/data-explorer'?

Here is the output from my prototype GPT model based on the issue.

main.bicep
@description('Deployment Location')
param location string

@description('Name of the Kusto Cluster. Must be unique within Azure.')
param name string = 'kusto${uniqueString(resourceGroup().id, subscription().id)}'

@description('The SKU of the Kusto Cluster.')
param sku string = 'Dev(No SLA)_Standard_D11_v2'

@description('The number of nodes in the Kusto Cluster.')
param numberOfNodes int = 1

@description('The version of the Kusto Cluster.')
param version string = 'latest'

@description('The tags of the Kusto Cluster.')
param tags object = {}

resource kustoCluster 'Microsoft.Kusto/clusters@2020-02-15' = {
  name: name
  location: location
  sku: {
    name: sku
  }
  properties: {
    numberOfNodes: numberOfNodes
    version: version
    tags: tags
  }
}

@description('The ID of the created or existing Kusto Cluster. Use this ID to reference the Kusto Cluster in other Azure resource deployments.')
output id string = kustoCluster.id
Summary

This Bicep module creates a Kusto Cluster with the specified number of nodes and version.

Description

Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more. Azure Data Explorer clusters are the fundamental resource in Azure Data Explorer. A cluster is a collection of compute resources that host databases and tables.

This Bicep module allows users to create or use existing Kusto Clusters with options to control the number of nodes and version. The output of the module is the ID of the created or existing Kusto Cluster, which can be used in other Azure resource deployments.

Examples

Example 1

This example creates a new Kusto Cluster with a unique name in the East US region using the default Kusto Cluster configuration settings. The module output is the ID of the created Kusto Cluster, which can be used in other Azure resource deployments.
module kustoCluster 'br/public:data-analytics/kusto-clusters:0.0.1' = {
  name: 'mykustocluster'
  params: {
    location: 'eastus'
  }
}

output kustoClusterID string = kustoCluster.outputs.id
Example 2

This example creates a new Kusto Cluster with the name "mykustocluster" in the "myresourcegroup" resource group located in the East US region. The Kusto Cluster is configured to use a specific version and number of nodes. The module output is the ID of the created or existing Kusto Cluster, which can be used in other Azure resource deployments.
param location string = 'eastus'
param name string = 'mykustocluster'
param resourceGroupName string = 'myresourcegroup'
param version string = 'latest'
param numberOfNodes int = 1

module kustoCluster 'br/public:data-analytics/kusto-clusters:0.0.1' = {
  name: 'mykustocluster'
  params: {
    location: location
    name: name
    resourceGroupName: resourceGroupName
    version: version
    numberOfNodes: numberOfNodes
  }
}

output kustoClusterID string = kustoCluster.outputs.id
test.bicep
/*
Write deployment tests in this file. Any module that references the main
module file is a deployment test. Make sure at least one test is added.
*/

param location string = resourceGroup().location

//Prerequisites
// module prereq 'prereq.test.bicep' = {
//   name: 'test-prereqs'
//   params: {
//     location: location
//   }
// }

//Test 0. 
module test0 '../main.bicep' = {
  name: 'test0'
  params: {
    location: location
  }
}

above looks nice hehe, agree! we can go with the storage/data-explorer path for module but waiting for proposal approval before start work on this module.

shenglol · 2023-04-08T03:29:11Z

On second thought, what about changing the path to data/data-explorer? storage makes me think that it's related to Microsoft.Storage.

vlahane · 2023-04-08T03:38:41Z

should we go with kusto/data-explorer ?

shenglol · 2023-04-08T04:15:27Z

Actually kusto might be even better since that matches the RP name. @dciborow thoughts?

vlahane · 2023-04-08T05:13:58Z

@shenglol should we also plan for other module to move to their respective RP groups like compute/container-registry rename it to container-registry/registry or container-registry/acr ?

vlahane · 2023-04-08T05:33:50Z

Hi @shenglol @dciborow Should we follow below naming for all of our module, any thoughts?

https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/resource-abbreviations

Syntax:

category + abbreviation

Example:

Resource	Old Path	New Path
Container registry	compute/container-registry	containers/cr
Azure Data Explorer cluster	storage/data-explorer	analytic-iot/dec
Virtual network	network/virtual-network	networking/vnet
Availability set	compute/availability-set	compute-web/avail
Function app	compute/function-app	compute-web/func

alex-frankel · 2023-04-10T15:47:39Z

Renaming modules is a breaking change, so I would prefer we keep the existing convention for now. compute is meant to be a category for all variety of computing services including resources like ACI or Functions.

shenglol · 2023-04-10T21:48:00Z

We never had an official discussion about the naming convention the module groups, but after reading @alex-frankel's comment in the Cosmos DB module PR, I start to realize that it makes sense to let module group names be at a higher level of abstraction and match the industry standard terms people use when dealing with cloud architecture. I'll make sure to document that in CONTRIBUTE.md and the issue template. I think we can keep the group name storage for this module.

Regarding changing group names for existing modules, like @alex-frankel said, it should be avoided because it's a breaking change, plus the fact that we don't have a way to mark modules as deprecated.

alex-frankel · 2023-04-10T23:13:51Z

plus the fact that we don't have a way to mark modules as deprecated.

Related to Azure/bicep#3505

vlahane · 2023-04-12T11:33:52Z

looks like OCI spec don't supporting yet, proposal still open "OCI Image Deprecation: vendor support."
https://github.com/opencontainers/image-spec/pull/903

## Description closes: #287  If you haven't already, read the full [contribution guide](https://github.com/Azure/bicep-registry-modules/blob/main/CONTRIBUTING.md). The guide may have changed since the last time you read it, so please double-check. Once you are done and ready to submit your PR, edit the PR description and run through the relevant checklist below. Enable GitHub Worksflows in your fork to enable auto-generation of assets with our [GitHub Action](/.github/workflows/push-auto-generate.yml). To trigger GitHub Actions after auto-generation, [add a GitHub PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) as a secret in your forked repository called `PAT`. ## Adding a new module  - [x] A proposal has been submitted and approved. - [x] I have included "Closes #{module_proposal_issue_number}" in the PR description. - [x] I have run `brm validate` locally to verify the module files. - [x] I have run deployment tests locally to ensure the module is deployable. --------- Co-authored-by: Daniel Ciborowski <[email protected]> Co-authored-by: dciborow <[email protected]> Co-authored-by: vlahane <[email protected]>

vlahane added the Module Proposal label Mar 23, 2023

shenglol added the Approved label Apr 5, 2023

This was referenced Apr 7, 2023

feat(new): Azure Data Explorer #308

Closed

[Module Collection]: Dedicated Azure Resource Bicep Modules #281

Closed

vlahane mentioned this issue Jul 6, 2023

feat(new): Azure Data Explorer (Kusto) #440

Merged

4 tasks

shenglol closed this as completed in #440 Aug 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Module Proposal]: Azure Data Explorer Clusters [Kusto/clusters] #287

[Module Proposal]: Azure Data Explorer Clusters [Kusto/clusters] #287

vlahane commented Mar 23, 2023 •

edited

Loading

dciborow commented Apr 5, 2023 •

edited

Loading

vlahane commented Apr 5, 2023

main.bicep

Summary

Description

Examples

Example 1

Example 2

test.bicep

shenglol commented Apr 8, 2023

vlahane commented Apr 8, 2023

shenglol commented Apr 8, 2023

vlahane commented Apr 8, 2023

vlahane commented Apr 8, 2023

alex-frankel commented Apr 10, 2023

shenglol commented Apr 10, 2023

alex-frankel commented Apr 10, 2023

vlahane commented Apr 12, 2023

[Module Proposal]: Azure Data Explorer Clusters [Kusto/clusters] #287

[Module Proposal]: Azure Data Explorer Clusters [Kusto/clusters] #287

Comments

vlahane commented Mar 23, 2023 • edited Loading

Have you checked this module does not already exist in the registry?

Why is the module needed?

Module path

Describe the module

dciborow commented Apr 5, 2023 • edited Loading

main.bicep

Summary

Description

Examples

Example 1

Example 2

test.bicep

vlahane commented Apr 5, 2023

main.bicep

Summary

Description

Examples

Example 1

Example 2

test.bicep

shenglol commented Apr 8, 2023

vlahane commented Apr 8, 2023

shenglol commented Apr 8, 2023

vlahane commented Apr 8, 2023

vlahane commented Apr 8, 2023

alex-frankel commented Apr 10, 2023

shenglol commented Apr 10, 2023

alex-frankel commented Apr 10, 2023

vlahane commented Apr 12, 2023

vlahane commented Mar 23, 2023 •

edited

Loading

dciborow commented Apr 5, 2023 •

edited

Loading