Prerequisites.md

Prerequisites

• Prerequisites
  • Tooling
  • Clone this Repo
  • Create Local Virtual Python Environment
  • Python Modules
  • Powershell Modules
  • Setup Environment Variables
  • Create Application Registration and Add to Environment Variables
  • Create Azure Dashboard

Tooling

You will need to ensure you have an Azure account with a subscription that you can use. You could use your MSDN subscription if you have one or you can sign up for a free trial using the link below.

• Azure Subscription

Please ensure you have the following tools installed on your machine.

• Visual Studio Code
• Azure CLI
• Bicep VS Code
• Bicep CLI
• Event Hub VS Code
• Powershell VS Code
• Python
  • Here is a good tutorial on how to install python along with spark and pyspark which is useful for data streaming pipelines.

Clone this Repo

To utilise the code in this repo, clone it from gihub and then open it up in Visual Studio code.

Create Local Virtual Python Environment

1. Open your Visual Studio code terminal.
2. Ensure you are in the root of the azure_eventdrivenarchitecture repo.
3. Run the following to create a virtual environment called venv.

python -m venv venv

Important Note : We will be using a single virtual environment for this entire implementation, but remember we are actually imitating a number of teams so the environment should actually be split per each teams implementation and what packages and dependencies they need along to carry out their function.

3. Activate the python environment by running the following.

venv\scripts\activate

4. To deactivate the python environment at any time run the following.

venv\scripts\deactivate

Python Modules

To complete all of the scenarios in this azure_eventdrivenarchitecture you need to activate your environment and install the following packages.

1. First activate your environment.

venv\scripts\activate

2. Then pip install the required packages.

pip install avro                       
pip install azure-common            
pip install azure-core           
pip install azure-cosmos             
pip install azure-eventhub        
pip install azure-functions         
pip install azure-identity            
pip install azure-keyvault-secrets  
pip install pip install azure-schemaregistry         
pip install azure-schemaregistry-avroserializer
pip install azure-storage-blob
pip install Faker

3. You can confirm the packages you have installed by running the following.

pip list

Remember: As mentioned above we are imitating a number of teams in this architecture and only creating one environment for all the packages for simplicity. In reality each team or implementation woud have only its own required libraries installed for its specific virtual environment.

Powershell Modules

1. You will need the suite of az powershell modules for some of the deployment exercises. To install these run the following in a powershell terminal.

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
Install-Module -Name Az -Scope CurrentUser -Repository PSGallery -Force

Setup Environment Variables

Throughout the course you will be utilising a number of variables repeatedly, and it is easier to store them in environment variables so that they can be referenced consistently.

They are:

• AZURE_UNIQUE_NAMESPACE - This will be the unique postfix you apply to all your Azure resources to ensure they are globally unique.
• AZURE_SUBSCRIPTION - This will be the name of the Azure subscription you are working in.
• AZURE_REGION - This will be the name of the Azure region you are working in.
• MY_IP - This is your client IP address.
• EXAMPLE_PASSWORD - This is a password value you can repeat across resources that require a password.

1. Open your system environment variables on your machine.
2. Add a unique namespace for your azure resources using the AZURE_UNIQUE_NAMESPACE variable name with an appropriate value. For example mine is "dgrf".
3. Add the name of the azure subscription you are going to be using the AZURE_SUBSCRIPTION as the variable name. For example mine is "dataGriff teaching".
4. Add the name of the azure subscription you are going to be using the AZURE_REGION variable name. For example mine is "northeurope".
5. Add your client IP address to a variable called MY_IP. You can get this value from whatismyipaddress.
6. Add an example password value t a variable called EXAMPLE_PASSWORD which contains upper and lower case characters, numbers and special chatacters.
7. Your system environment variables should now look something like this.
8. Once you have set your environment variable you will need to restart your IDE (Visual Studio Code) to pick up the new values.

Throughout the code these system variables will be referenced like the examples below.

Like this in Powershell...

[System.Environment]::GetEnvironmentVariable('AZURE_UNIQUE_NAMESPACE')

Like this in Python...

unique_namespace = os.environ.get('AZURE_UNIQUE_NAMESPACE')

Like this in command line...

"%AZURE_UNIQUE_NAMESPACE%"

Important Help - When using the environment variables and switching languages, your values may get in a bit of a twist on occassion. If you do get any errors I recommend just killing all terminals, restarting visual studio code and starting from scratch.

Create Application Registration and Add to Environment Variables

We are going to be creating an application registration that we will use for the majority of our authentication against our new estate for the following exercises. We will also store these credentials in our system environment variables.

1. In the Azure Portal go into application registrations in Azure Active directory (you can also find a link to this on your dashboard in the markdown on the left).
2. Create a new app registration called aprg-events-admin.

We are now going to store the credentials of the application registration created above in our local environment variables. These values will be used whenever we reference the below sections of code in our python scripts as per this online documentation.

from azure.identity._credentials.default import DefaultAzureCredential
...
token_credential = DefaultAzureCredential()

1. Edit the environment variables for your account on your local machine.
2. First add a variable for AZURE_CLIENT_ID from your new application registration.
3. Then add a variable for AZURE_TENANT_ID from your new application registration.
4. Next we need to generate a secret from our application registration and add this to our environment variables.
5. Once you have copied this value, paste it into a new local environment variable called AZURE_CLIENT_SECRET.

Create Azure Dashboard

1. Copy and pase the dashboard.json file and call it mydashboard.json. This will be your own copy of an Azure dashboard that will make it easier to navigate all the resources in this course.

2. Find and replace the {primaryDomain} text pattern in mydashboard.json with your Azure primary domain. You can get this from Azure Active directory blade in the portal. If you are unsure where to find Azure Active directory, just type "Azure Active Directory" into the search bar at the top of the portal.

3. Find and replace the {primaryDomain} text pattern in mydashboard.json with your Azure primary domain. You can get this from the subscription blade in the portal. If you are unsure where to find your subscription, just type "subscription" into the search bar at the top of the portal and then select your subscription.

4. Upload your dashboard to the portal by selecting dashboard from the left hand menu and then select new dashboard and upload. Choose your mydashboard.json file.

5. Your dashboard should look something like the below as the resource groups and resources have not been deployed yet.