ML.NET 5.0.0-preview1

ML.NET 5.0-preview.1

New Features

• Introducing SentencePiece Unigram Tokenizer Model (#7390)
• Phi-4 Tokenizer Support (#7396)

Enhancements

• Support O3 OpenAI model mapping (#7394)

Bug Fixes

• Fix DateTime export to csv with culture info (#7358) - Thanks @asmirnov82!
• Moved SpecialTokens assignment after the modification to avoid "Collection Modified" error (#7328) - Thanks @shaltielshmid!
• Fix broken inheritance from DataFrameColumn class (#7324) - Thanks @asmirnov82!

Build / Test updates

• [main] Update dependencies from dotnet/arcade (#7387)
• Update Microsoft.Extensions.AI to 9.3.0-preview.1.25114.11 (#7388)
• Update MSTest to latest (#7349)
• [main] Update dependencies from dotnet/arcade (#7352)
• Consolidate System.Numerics.Tensors dependency (#7356) - Thanks @asmirnov82!
• [main] Update dependencies from dotnet/arcade (#7382)
• [main] Update dependencies from dotnet/arcade (#7376)
• [main] Update dependencies from dotnet/arcade (#7374)
• Net8 tests (#7319)
• [main] Update dependencies from dotnet/arcade (#7266)
• Update DependencyModel (#7338)
• Update M.E.AI version used by Microsoft.ML.GenAI.Core (#7329)
• Update System.Numerics.Tensors version (#7322) - Thanks @asmirnov82!

Documentation Updates

• Fix up docs for MLContext (#7334)
• Some tweaks to the Microsoft.ML.Tokenizers PACKAGE.md (#7360)
• [GenAI] SFT Example (#7316)
• Update dynamic loading report reference (#7321) - Thanks @emmanuel-ferdman!

ML.NET 4.0.2

ML.NET 4.0.2

Enhancements

• [release/4.0] Support O3 OpenAI model mapping #7395 (#7395)

Build / Test updates

• [release/4.0] Update dependencies from dotnet/arcade (#7369)
• [release/4.0] Update dependencies from dotnet/arcade (#7377)
• [release/4.0] Update dependencies from dotnet/arcade (#7386)

ML.NET 4.0.1

ML.NET Servicing Release 4.0.1

Bug Fixes

• Fix the BERT tokenizer to handle special tokens correctly. (#7330) - Thanks @shaltielshmid
• Fix broken inheritance from DataFrameColumn class (#7324) (#7331) - Thanks @asmirnov82!

Build / Test updates

• Update System.Numerics.Tensors version (#7322) (#7355) - Thanks @asmirnov82!
• [release/4.0] Update dependencies from dotnet/arcade (#7327)
• Update MicrosoftExtensionsDependencyModelVersion (#7342)

Documentation Updates

• [release/4.0] Some tweaks to the Microsoft.ML.Tokenizers PACKAGE.md (#7364)
• [release/4.0] Fix up docs for MLContext (#7363)

v4.0.0-preview1

What's Changed

• Migrate to the 'locker' GitHub action for locking closed/stale issues/PRs by @jeffhandley in #6896
• Update release-3.0.0.md by @taeerhebend in #6895
• Add support for Apache.Arrow.Types.TimestampType to DataFrame by @asmirnov82 in #6871
• Update developer-guide.md by @computerscienceiscool in #6870
• Issue #6606 - Add sample variance and standard deviation to NormalizeMeanVariance by @tearlant in #6885
• Reorganize dataframe files by @asmirnov82 in #6872
• Updated ml.net versioning by @michaelgsharp in #6907
• Don't include the SDK in our helix payload by @ericstj in #6918
• Rename NameEntity to NamedEntity by @ericstj in #6917
• Fix assert by only accessing idx by @ericstj in #6924
• Make double assertions compare with tolerance instead of precision by @ericstj in #6923
• [main] Update dependencies from dotnet/arcade by @dotnet-maestro in #6703
• Only use semi-colons for NoWarn - fixes build break by @ericstj in #6935
• Fixes NER to correctly expand/shrink the labels by @michaelgsharp in #6928
• Split out non concurrent test collections. by @michaelgsharp in #6937
• Add Backport github workflow by @ericstj in #6944
• fix #6949 by @LittleLittleCloud in #6951
• Torch sharp version updates and test fixes by @michaelgsharp in #6954
• [main] Update dependencies from dotnet/arcade by @dotnet-maestro in #6957
• Added in release notes for 3.0.1 by @michaelgsharp in #6962
• Update .NET Runtimes to latest version by @ericstj in #6964
• Add sweepable estimator to NER by @zewditu in #6965
• Testing light gbm bad allocation by @michaelgsharp in #6968
• [main] Update dependencies from dotnet/arcade by @dotnet-maestro in #6969
• Add new type to key-value converter by @zewditu in #6973
• [main] Update dependencies from dotnet/arcade by @dotnet-maestro in #6976
• Introducing Tiktoken Tokenizer by @tarekgh in #6981
• [main] Update dependencies from dotnet/arcade by @dotnet-maestro in #6985
• FabricBot: Onboarding to GitOps.ResourceManagement because of FabricBot decommissioning by @dotnet-policy-service in #6983
• Add text normalizer transformer to AutoML by @zewditu in #6998
• Tokenizer's Interfaces Cleanup by @tarekgh in #7001
• First round of perf improvements for tiktoken by @stephentoub in #7012
• Avoid LruCache in Tiktoken when cacheSize specified is 0 by @stephentoub in #7016
• Tweak Tiktoken's BytePairEncode for improved perf by @stephentoub in #7017
• Tweak CreateByModelNameAsync by @stephentoub in #7015
• Match SweepableEstimatorFactory name with Ml.net name. by @zewditu in #7007
• Update OnnxRuntime to 1.16.3 by @michaelgsharp in #6975
• Temp fix for the race condition during the tests. by @michaelgsharp in #7021
• Fix formatting that fails in VS by @michaelgsharp in #7023
• Optimize regexes used in tiktoken by @stephentoub in #7020
• Address the feedback on the tokenizer's library by @tarekgh in #7024
• Make MlImage tests not block file for reading by @michaelgsharp in #7029
• Packaging cleanup by @ericstj in #6939
• Remove SourceLink SDK references by @ericstj in #7037
• Add Span support in tokenizer's Model abstraction by @tarekgh in #7035
• [main] Update dependencies from dotnet/arcade by @dotnet-maestro in #6995
• Change official build to use 1ES templates by @directhex in #7048
• Adding needed Tokenizer's APIs by @tarekgh in #7047
• Auto-generated baselines by 1ES Pipeline Templates by @directhex in #7051
• testing arm python brew overwite by @michaelgsharp in #7058
• Update package versions in use by ML.NET tests by @ericstj in #7055

New Contributors

• @taeerhebend made their first contribution in #6895
• @computerscienceiscool made their first contribution in #6870
• @tearlant made their first contribution in #6885
• @dotnet-policy-service made their first contribution in #6983
• @directhex made their first contribution in #7048

Full Changelog: v3.0.1...v4.0.0-preview1

ML.NET 3.0.1

ML.NET 3.0.1

New Features

• Add support for Apache.Arrow.Types.TimestampType to DataFrame (#6871) - Thanks @asmirnov82!

Enhancements

• Update TorchSharp to latest version (#6954)
• Reorganize dataframe files (#6872) - Thanks @asmirnov82!
• Add sample variance and standard deviation to NormalizeMeanVariance (#6885) - Thanks @tearlant!
• Fixes NER to correctly expand/shrink the labels (#6928)

Bug Fixes

• Fix SearchSpace reference not being included (#6951)
• Rename NameEntity to NamedEntity (#6917)
• Fix assert by only accessing idx (#6924)

Build / Test updates

• Add Backport github workflow (#6944)
• Branding for 3.0.1 (#6943)
• Only use semi-colons for NoWarn - fixes build break (#6935)
• Update dependencies from dotnet/arcade (#6703)
• Update dependencies from dotnet/arcade (#6957)
• Migrate to the 'locker' GitHub action for locking closed/stale issues/PRs (#6896)
• Make double assertions compare with tolerance instead of precision (#6923)
• Don't include the SDK in our helix payload (#6918)

Documentation Updates

• Updated ml.net versioning (#6907)
• Update developer-guide.md (#6870) - Thanks @computerscienceiscool!
• Update release-3.0.0.md (#6895) - Thanks @taeerhebend!

Breaking changes

• Rename NameEntity to NamedEntity (#6917)

ML.NET 3.0.0

ML.NET 3.0.0

New Features

• Add the ability to use Object Detection using TorchSharp (#6605) - We have added a new deep learning model back by TorchSharp that lets you fine tune your own Object Detection model!
• Add SamplingKeyColumnName to AutoMLExperiment API (#6649) - You can now set the SamplingKeyColumnName when you are using AutoML. Thanks @torronen!
• Add Object Detection to AutoML Sweeper (#6633) - Added Object Detection to the AutoML Sweeper so now they can be used together.
• Add String Vector support to DataFrame (#6628) - Adds support for String Vectors in DataFrame. This also allows for Better IDataView <-> DataFrame conversions.
• Add AutoZero tuner to BinaryClassification (#6615) - Can now use AutoZero tuner in AutoML Binary Classification experiments.
• Added in fairness assessment and mitigation (#6539) - Support for fairness assessment and mitigation tool
• Added in Support for some Intel OneDal Algorithms (#6521) - You can now use Intel's OneDal for some algorithms. This gives you access to some accelerated versions of these algorithms. The models are fully interoperable between ML.NET's normal models and these, so you can train with OneDal and then still run on machines where OneDal is not supported. Thanks @rgesteve!
• Add in ability to have pre-defined weights for ngrams (#6458) - If you know the weights of your NGrams already you can now directly provide that.
• Add SentenceSimilarity sweepable estimator in AutoML (#6445) - Can now use SentenceSimilarity with the sweepable estimator.
• Add VBufferDataFrameCoumn to DataFrame (#6409) - Now DataFrame can support the VBuffer from ML.NET so the IDataView <-> DataFrame conversion can work with those types.
• Added ADO.NET importing/exporting functionality to DataFrame (#5975) - Can now use ADO.NET import/export with DataFrames. Thanks @andrei-faber!
• Added native binaries for Windows Arm64 (#6813) - This allows certain native transforms to be run on Widows Arm that were disabled before.
• Switches some computational code to use the new Tensor Primitives package (#6875)
• Add QA sweepable estimator in AutoML (#6781)
• Add NameEntityRecognition and Q&A deep learning tasks. (#6760)
• Adds the ability to load a pre-trained LightGBM file and import it into ML.Net. (#6569)

Enhancements

• Expose ExperimentSettings.MaxModel as public (#6663) - Exposes ExperimentSettings.MaxModel as public so now you can set the number of Max Models you want for an AutoML experiment.
• Update to latest version of TorchSharp (#6636) - Updated to the latest version of TorchSharp and fixed any breaking changes so we can take advantage of their new features and bug fixes.
• Update to latest version of Onnx Runtime (#6624) - Updated to the latest version of Onnx Runtime and fixed any breaking changes so we can take advantage of their new features and bug fixes.
• Update ML.NET to compile with .NET8 (#6641) - Removed some deprecated code now throws errors on .NET8 as well as other minor fixes to allow working/building with .NET8.
• Added more logging to Object Detection (#6646) - Added more logging while Object Detection is training so even if epochs take a long time you can be sure things are still moving.
• Update timeout error message in AutoMLExperiment (#6613) - Updated the error message so it is more clear what happened.
• Add batchsize and arch to imageClassification SweepableTrainer (#6597) - Added batchsize and arch to the ImageClassification SweepableTrainer so those can now be trained on.
• Update max_model when trial fails (#6596)
• Add default search space for standard trainers (#6576) - Added a default search space for all standard trainers so users have reasonable default values.
• Adding more metrics to BinaryClassification Experiment (#6571)
• Add checkAlive in NasBertTrainer (#6546) - Now we check between batches if cancellation was requested and stop processing if so.
• OneDAL - Fallback to default implementation (#6538) - If you specify you want to use OneDal but something happens that prevents you from using it, like it can't find the binaries/etc, it will auto default back to the normal implementation instead of crashing.
• Add addKeyValueAnnotationsAsText flag in AutoML (#6535)
• Add continuous resource monitoring to AutoML.IMonitor (#6520) - Thanks @andrasfuchs!
• Update WebClient to HttpClient implementations (#6476) - Update the usage of WebClient to HttpClient since WebClient is now deprecated. Thanks @rgesteve!
• Set AutoML trial to unsuccess if trial loss is nan/inf (#6430) - Now trial will be marked as unsuccesssful if the loss is an invalid number.
• Add diskConvert option in fast tree search space (#6316)
• Avoid Boxing/Unboxing on accessing elements of VBufferDataFrameColumn (#6867) and (#6865) - Thanks @asmirnov82!
• Update LightGBM to version 3.X.X from 2.X.X (#6880)
• Implement vectorized binary arithmetic operations for DataFrames (#6854) - Thanks @asmirnov82!
• Upgrade .NET Interactive (#6857) - Thanks @colombod!
• Improve performance of column cloning inside DataFrame arithmetics (#6814) - Thanks @asmirnov82!
• Add performance benchmarks for dataframe arithmetic operations (#6827) - Thanks @asmirnov82!
• Simplify tt files for PrimitiveDataFrameColumnAritmetics (#6830) - Thanks @asmirnov82!
• Improve performance of DataFrame binary comparison operations (#6869) - Thanks @asmirnov82!
• Allow a CultureInfo to be used for parsing CSV values into DataFrame (#6782) - Thanks @asmirnov82!
• File-scoped namespaces in files under Prediction (Microsoft.ML.Core) (#6792) - Thanks @Lehonti!
• File-scoped namespaces in files under ComponentModel (Microsoft.ML.Core) (#6788) - Thanks @Lehonti!
• File-scoped namespaces in files under Data (Microsoft.ML.Core) (#6789) - Thanks @Lehonti!
• File-scoped namespaces in files under EntryPoints (Microsoft.ML.Core) (#6790) - Thanks @Lehonti!
• File-scoped namespaces in files under Environment (Microsoft.ML.Core) (#6791) - Thanks @Lehonti!
• Add TargetType to Type_convert (#6785)
• Modernized some argument checks that still used string literals for parameter names (#6766) - Thanks @Lehonti!
• Improve DataFrame Arithmetics implementation (#6763) - Thanks @asmirnov82!
• Fixed mac build and minor torch sharp changes (#6776)
• Clean DataFrame meaningless code (#6761) - Thanks @asmirnov82!
• Provide ability to filter dataframe column by null via ElementWise Methods (#6723)
• Add missing implementation for datetime relevant arrow type into dataframe (#6675) - Thanks @asmirnov82!
• Fix DataFrame to allow to store columns with size more than 2 Gb (#6710) - Thanks @A...

ML.NET 3.0.0 Preview 2

ML.NET 3.0.0 Preview

New Features

• Add the ability to use Object Detection using TorchSharp (#6605) - We have added a new deep learning model back by TorchSharp that lets you fine tune your own Object Detection model!
• Add SamplingKeyColumnName to AutoMLExperiment API (#6649) - You can now set the SamplingKeyColumnName when you are using AutoML. Thanks @torronen!
• Add Object Detection to AutoML Sweeper (#6633) - Added Object Detection to the AutoML Sweeper so now they can be used together.
• Add String Vector support to DataFrame (#6628) - Adds support for String Vectors in DataFrame. This also allows for Better IDataView <-> DataFrame conversions.
• Add AutoZero tuner to BinaryClassification (#6615) - Can now user AutoZero tuner in AutoML Binary Classification experiments.
• Added in fairness assessment and mitigation (#6539) - Support for fairness assessment and mitigation tool
• Added in Support for some Intel OneDal Algorithms (#6521) - You can now use Intel's OneDal for some algorithms. This gives you access to some accelerated versions of these algorithms. The models are fully interoperable between ML.NET's normal models and these, so you can train with OneDal and then still run on machines where OneDal is not supported. Thanks @rgesteve!
• Add in ability to have pre-defined weights for ngrams (#6458) - If you know the weights of your NGrams already you can now directly provide that.
• Add SentenceSimilarity sweepable estimator in AutoML (#6445) - Can now use SentenceSimilarity with the sweepable estimator.
• Add VBufferDataFrameCoumn to DataFrame (#6445) - Now DataFrame can support the VBuffer from ML.NET so the IDataView <-> DataFrame conversion can work with those types.
• Added ADO.NET importing/exporting functionality to DataFrame (#5975) - Can now use ADO.NET import/export with DataFrames. Thanks @andrei-faber!

Enhancements

• Expose ExperimentSettings.MaxModel as public (#6663) - Exposes ExperimentSettings.MaxModel as public so now you can set the number of Max Models you want for an AutoML experiment.
• Update to latest version of TorchSharp (#6636) - Updated to the latest version of TorchSharp and fixed any breaking changes so we can take advantage of their new features and bug fixes.
• Update to latest version of Onnx Runtime (#6624) - Updated to the latest version of Onnx Runtime and fixed any breaking changes so we can take advantage of their new features and bug fixes.
• Update ML.NET to compile with .NET8 (#6641) - Removed some deprecated code now throws errors on .NET8 as well as other minor fixes to allow working/building with .NET8.
• Added more logging to Object Detection (#6646) - Added more logging while Object Detection is training so even if epochs take a long time you can be sure things are still moving.
• Update timeout error message in AutoMLExperiment (#6613) - Updated the error message so it is more clear what happened.
• Add batchsize and arch to imageClassification SweepableTrainer (#6597) - Added batchsize and arch to the ImageClassification SweepableTrainer so those can now be trained on.
• Update max_model when trial fails (#6596)
• Add default search space for standard trainers (#6576) - Added a default search space for all standard trainers so users have reasonable default values.
• Adding more metrics to BinaryClassification Experiment (#6571)
• Add checkAlive in NasBertTrainer (#6546) - Now we check between batches if cancellation was requested and stop processing if so.
• OneDAL - Fallback to default implementation (#6538) - If you specify you want to use OneDal but something happens that prevents you from using it, like it can't find the binaries/etc, it will auto default back to the normal implementation instead of crashing.
• Add addKeyValueAnnotationsAsText flag in AutoML (#6535)
• Add continuous resource monitoring to AutoML.IMonitor (#6520) - Thanks @andrasfuchs!
• Update WebClient to HttpClient implementations (#6476) - Update a usage of WebClient to HttpClient since WebClient is now deprecated. Thanks @rgesteve!
• Set AutoML trial to unsuccess if trial loss is nan/inf (#6430) - Now trial will be marked as unsuccesssful if the loss is an invalid number.
• Add diskConvert option in fast tree search space (#6316)

Bug Fixes

• Fix DataFrame ToString (#6673) - Use correct alignment for columns to produce readable output when columns have longer names. Thanks @asmirnov82!
• Fix DataFrame null math (#6661) - Fixes max in DataFrame columns when there are null values to match what Pandas does.
• Clean up PrimitiveColumnContainer (#6656) - Cleaned up the code in PrimitiveColumnContainer so its more correct and easier to use.
• Fix Apply in PrimitiveColumnContainer (#6642) - Fixes the Apply method so it no longer changes the source column. Thanks @janholo!
• Fix datetime null error (#6627) - Fixes loading a null datetime from a database so it now returns correctly instead of throwing an error.
• Fix AggregateTrainingStopManager is trying to cancel disposed tokens (#6612) - Will no longer try and cancel already disposed tokens.
• Fix tostring bug for sweepable pipeline (#6610)
• Change Test to Validate in Dataset manager (#6599)
• Fixed System.OperationCanceledException when calling experimentResult.BestRun.Estimator.Fit (#6572)
• Fixed cancellation bug in SweepablePipelineRunner && Fixed object null exception in AutoML v1.0 regression API (#6560)
• Fixed one dal dispatching issues (#6547) - OneDal now dispatches correctly.
• Fixed Multi-threaded access issue (#6537) - Fixed a multi-threaded access issue for variable length string arrays in ONNX models.
• Fixed AutoML experiments in non declarative style not working (#6447)

Build / Test updates

• Remove MSIL Check for TorchSharp (#6658) - Removes the MSIL check for TorchSharp while we figure out how we want to correctly handle this.
• Change code coverage build pool (#6647) - Changed codecoverage build pool so the builds are faster and more stable.
• Update AutoMLExperimentTests.cs to fix timeout error (#6638)
• Update FabricBot config (#6619)
• Libraries area pod updates March 2023 (#6607)
• Update dependencies from dotnet/arcade (#6566 & #6518 & #6451 & #6439)
• Mac python fix (#6549)
• Moving onedal nuget download from onedal to native where its needed for building (#6527)
• New os image for official builds (#6467)

Documentation Updates

• Add doc for CreateSweepableEstimator, Parameter and SearchSpace (#6611)
• Add AutoMLExperiment example doc (#6594)
• Fix minor doc typos (#6557)
• Fix minor roadmap nits (#6480)
• 2023 roadmap outline (#6444)
• Fixed typo for calibrators ([#6438](https://github.com/do...

ML.NET 2.0.1 Preview

Minor update for 2.0.0 that introduces a new api for ProduceWordBag so you can pass in a column that already has the weights set.

ML.NET 1.7.1

Minor servicing update with dependency updates and PFI bug fix for correctly finding the correct transformer to use.

ML.NET 1.7.0 RC 1

ML.NET 1.7.0 RC 1

Moving forward, we are going to be aligning more with the overall .NET release schedule. As such, this is a smaller release since we had a larger one just about 3 months ago but it aligns us with the release of .NET 6.

New Features

ML.NET

• Switched to getting version from assembly custom attributes- (#4512) Remove reliance on getting product version for model.zip/version.txt from FileVersionInfo and replace with using assembly custom attributes. This will help in supporting single file applications. (Thanks @r0ss88)
• Can now optionally not dispose of the underlying model when you dispose a prediction engine. (#5964) A new prediction engine options class has been added that lets you determine if the underlying model should be disposed of or not when the prediction engine itself is disposed of.
• Can now set the number of threads that onnx runtime uses (#5962) This lets you specify the number of parallel threads ONNX runtime will use to execute the graph and run the model. (Thanks @yaeldekel)
• The PFI API has been completely reworked and is now much more user friendly (#5934) You can now get the output from PFI as a dictionary mapping the column name (or the slot name) to its PFI result.

DataFrame

• Can now merge using multiple columns in a JOIN condition (#5838) (Thanks @asmirnov82)

Enhancements

ML.NET

• Run formatting on all src projects (#5937) (Thanks @jwood803)
• Added BufferedStream for reading from DeflateStream - reduces loading time for .NET core (#5924) (Thanks @martintomasek)
• Update editor config to match Roslyn and format samples (#5893) (Thanks @jwood803)
• Few more minor editor config changes (#5933)

DataFrame

• Use Equals and = operator for DataViewType comparison (#5942) (Thanks @thoron)

Bug Fixes

• Initialize _bestMetricValue when using the Loss metric (#5939) (Thanks @MiroslavKabat)

Build / Test updates

• Changed the queues used for building/testing from Ubuntu 16.04 to 18.04 (#5970)
• Add in support for building with VS 2022. (#5956)
• Codecov yml token was added (#5950)
• Move from XliffTasks to Microsoft.DotNet.XliffTasks (#5887)

Documentation Updates

• Fixed up Readme, updated the roadmap, and new doc detailing some platform limitations. (#5892)

Breaking Changes

• None