Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V2 Common MLOps pipelines for AutoML & CustomScript #105

Draft
wants to merge 38 commits into
base: v2-preview
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
5b3c1b4
Move mlops-pipelines folder to root
mariamedp Sep 18, 2020
b834ace
Setup pipeline
mariamedp Sep 18, 2020
f3e38eb
Build training pipeline using configuration files
mariamedp Sep 23, 2020
4622d5b
Build forecasting pipeline using configuration files
mariamedp Sep 23, 2020
449405c
Modeling pipeline - update data & invoke training pipeline
mariamedp Sep 23, 2020
9bdc199
Customscript train params in script settings file
mariamedp Sep 24, 2020
4eda3ab
Customscript forecast params in settings file
mariamedp Sep 24, 2020
e42b772
Update batch forecasting pipeline & customize data update for trainin…
mariamedp Sep 24, 2020
8daf6e1
Adapt realtime deployment
mariamedp Sep 25, 2020
aedec46
Adapt customscript notebooks
mariamedp Sep 28, 2020
d8ea30e
Rename pipelines
mariamedp Sep 29, 2020
5963c5b
Update MLOps README
mariamedp Oct 1, 2020
08cfc36
Small fixes & update param in model deployment to update all webservi…
mariamedp Oct 6, 2020
b5e8d25
Add UPDATE_DEPLOYMENT variable in ADO vg
mariamedp Oct 6, 2020
01e6592
Incorporate changes from 'v2-preview'
mariamedp Oct 6, 2020
4c33df6
Option to avoid deploying router webservice
mariamedp Oct 7, 2020
1567fe8
Address notebooks and mlops-pipelines folders in documentation
mariamedp Oct 8, 2020
3611c6b
Publish AML pipeline before invoking in same pipeline
mariamedp Oct 8, 2020
a0930d1
Fix deployment for 1 single model
mariamedp Oct 9, 2020
1bd66c9
Small fixes in deployment
mariamedp Oct 9, 2020
7756384
Batch forecasting pipeline artifact
mariamedp Oct 14, 2020
0ad2303
Change params by vars in mlops pipelines and reestructure a bit
mariamedp Oct 14, 2020
079564c
Add extensive documentation for MLOps pipelines
mariamedp Oct 15, 2020
4db384a
Split READMEs and add Details pages, fix typos and links
mariamedp Oct 15, 2020
95f5b44
Fix typos and links in the documentation
mariamedp Oct 15, 2020
182e74c
Update flow map
mariamedp Oct 16, 2020
5df84ee
Merge branch 'v2/mlopscommon' into v2/documentation
mariamedp Oct 16, 2020
3344604
Add validation for PRS config
mariamedp Oct 16, 2020
235eeef
Container concurrency disabled by default
mariamedp Oct 16, 2020
c9952f3
Rename forecast.py to batch_forecasting.py
mariamedp Oct 16, 2020
c46fd3c
Merge pull request #107 from microsoft/v2/documentation
mariamedp Nov 9, 2020
14cc78c
Merge after closing doc PR
mariamedp Nov 17, 2020
4002543
MLOps pipelines images
mariamedp Nov 17, 2020
ab6897d
Move AutoML notebooks
mariamedp Jan 14, 2021
b6eacfd
Pip install param
mariamedp Jan 14, 2021
f3477f3
First automl mlops pipelines cleanup
mariamedp Jan 14, 2021
d95abe2
Move automl training script
mariamedp Jan 14, 2021
6dd1cb6
Adapt automl training script
mariamedp Jan 14, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 14 additions & 12 deletions .github/workflows/custom-script-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ on:
branches:
- v2-preview
paths:
- 00_Setup_AML_Workspace.ipynb
- 01_Data_Preparation.ipynb
- Custom_Script/**
- notebooks/00_Setup_AML_Workspace.ipynb
- notebooks/01_Data_Preparation.ipynb
- notebooks/Custom_Script/**
env:
SUBSCRIPTION_ID: ${{ secrets.SUBSCRIPTION_ID }}
RESOURCE_GROUP: ${{ secrets.RESOURCE_GROUP }}
Expand All @@ -34,13 +34,13 @@ jobs:
architecture: 'x64'
- name: Install dependencies
run: |
python -m pip install -r Custom_Script/requirements.txt
python -m pip install -r notebooks/Custom_Script/requirements.txt
- name: Convert notebooks to python
run: |
jupyter nbconvert --to script 00_Setup_AML_Workspace.ipynb
jupyter nbconvert --to script 01_Data_Preparation.ipynb
jupyter nbconvert --to script Custom_Script/02_CustomScript_Training_Pipeline.ipynb
jupyter nbconvert --to script Custom_Script/03_CustomScript_Forecasting_Pipeline.ipynb
jupyter nbconvert --to script notebooks/00_Setup_AML_Workspace.ipynb
jupyter nbconvert --to script notebooks/01_Data_Preparation.ipynb
jupyter nbconvert --to script notebooks/Custom_Script/02_CustomScript_Training_Pipeline.ipynb
jupyter nbconvert --to script notebooks/Custom_Script/03_CustomScript_Forecasting_Pipeline.ipynb
- name: Login via Az module
uses: azure/[email protected]
with:
Expand All @@ -54,19 +54,21 @@ jobs:
echo "************************************"
echo "Using workspace: $Env:WORKSPACE_NAME"
echo "************************************"
cd notebooks
python 00_Setup_AML_Workspace.py
$resource = Get-AzResource -Name $Env:WORKSPACE_NAME -ResourceGroupName $Env:RESOURCE_GROUP
$expirationTime = (Get-Date).AddHours(24).ToUniversalTime()
New-AzTag -ResourceId $resource.id -Tag @{"expiresOn"=$expirationTime}
azPSVersion: 'latest'
- name: Data preparation
run: |
cd notebooks
ipython 01_Data_Preparation.py
- name: Run training
run: |
cd Custom_Script
python 02_CustomScript_Training_Pipeline.py
cd notebooks/Custom_Script
ipython 02_CustomScript_Training_Pipeline.py
- name: Run prediction
run: |
cd Custom_Script
python 03_CustomScript_Forecasting_Pipeline.py
cd notebooks/Custom_Script
ipython 03_CustomScript_Forecasting_Pipeline.py
File renamed without changes
Binary file added .images/Flow_map.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
File renamed without changes
File renamed without changes
Binary file added .images/mlops_pipeline1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .images/mlops_pipeline2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .images/mlops_pipeline3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .images/mlops_pipeline4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .images/mlops_pipeline5.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .images/mlops_pipeline_1_setup.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .images/mlops_pipeline_2_modeling.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .images/mlops_pipeline_3_batchforecasting.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
File renamed without changes
File renamed without changes

This file was deleted.

This file was deleted.

60 changes: 0 additions & 60 deletions Automated_ML/mlops-pipelines/1-setup/setup-pipeline.yml

This file was deleted.

This file was deleted.

Loading