forked from gtbluesky/onnxruntime_flutter
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
4 changed files
with
78 additions
and
76 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,3 @@ | ||
## 0.0.1 | ||
## 0.0.2 | ||
|
||
* TODO: Describe initial release. | ||
* initial release. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,21 @@ | ||
TODO: Add your license here. | ||
MIT License | ||
|
||
Copyright (c) 2023 gtbluesky | ||
|
||
Permission is hereby granted, free of charge, to any person obtaining a copy | ||
of this software and associated documentation files (the "Software"), to deal | ||
in the Software without restriction, including without limitation the rights | ||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
copies of the Software, and to permit persons to whom the Software is | ||
furnished to do so, subject to the following conditions: | ||
|
||
The above copyright notice and this permission notice shall be included in all | ||
copies or substantial portions of the Software. | ||
|
||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
SOFTWARE. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,92 +1,74 @@ | ||
# onnxruntime | ||
<p align="center"><img width="50%" src="https://github.com/microsoft/onnxruntime/raw/main/docs/images/ONNX_Runtime_logo_dark.png" /></p> | ||
|
||
A new Flutter FFI plugin project. | ||
# OnnxRuntime Plugin | ||
[](https://pub.dev/packages/onnxruntime) | ||
|
||
## Getting Started | ||
|
||
This project is a starting point for a Flutter | ||
[FFI plugin](https://docs.flutter.dev/development/platform-integration/c-interop), | ||
a specialized package that includes native code directly invoked with Dart FFI. | ||
|
||
## Project structure | ||
|
||
This template uses the following structure: | ||
## Overview | ||
|
||
* `src`: Contains the native source code, and a CmakeFile.txt file for building | ||
that source code into a dynamic library. | ||
Flutter plugin for OnnxRuntime with `FFI` provides an easy, flexible, and fast Dart API to integrate Onnx models in flutter apps across mobile and desktop platforms. | ||
|
||
* `lib`: Contains the Dart code that defines the API of the plugin, and which | ||
calls into the native code using `dart:ffi`. | ||
## Key Features | ||
|
||
* platform folders (`android`, `ios`, `windows`, etc.): Contains the build files | ||
for building and bundling the native code library with the platform application. | ||
* Multi-platform Support for Android and iOS | ||
* Flexibility to use any Onnx Model. | ||
* Acceleration using multi-threading. | ||
* Similar structure as OnnxRuntime Java and C# API. | ||
* Inference speeds close to native Android/iOS Apps built using the Java/Objective-C API. | ||
* Run inference in different isolates to prevent jank in UI thread. | ||
|
||
## Building and bundling native code | ||
## Getting Started | ||
|
||
The `pubspec.yaml` specifies FFI plugins as follows: | ||
In your flutter project add the dependency: | ||
|
||
```yaml | ||
plugin: | ||
platforms: | ||
some_platform: | ||
ffiPlugin: true | ||
```yml | ||
dependencies: | ||
... | ||
onnxruntime: | ||
``` | ||
This configuration invokes the native build for the various target platforms | ||
and bundles the binaries in Flutter applications using these FFI plugins. | ||
This can be combined with dartPluginClass, such as when FFI is used for the | ||
implementation of one platform in a federated plugin: | ||
For help getting started with Flutter, view the online | ||
[documentation](https://flutter.io/). | ||
```yaml | ||
plugin: | ||
implements: some_other_plugin | ||
platforms: | ||
some_platform: | ||
dartPluginClass: SomeClass | ||
ffiPlugin: true | ||
``` | ||
## Usage example | ||
A plugin can have both FFI and method channels: | ||
### Import | ||
```yaml | ||
plugin: | ||
platforms: | ||
some_platform: | ||
pluginClass: SomeName | ||
ffiPlugin: true | ||
```dart | ||
import 'package:onnxruntime/onnxruntime.dart'; | ||
``` | ||
|
||
The native build systems that are invoked by FFI (and method channel) plugins are: | ||
### Initializing environment | ||
|
||
* For Android: Gradle, which invokes the Android NDK for native builds. | ||
* See the documentation in android/build.gradle. | ||
* For iOS and MacOS: Xcode, via CocoaPods. | ||
* See the documentation in ios/onnxruntime.podspec. | ||
* See the documentation in macos/onnxruntime.podspec. | ||
* For Linux and Windows: CMake. | ||
* See the documentation in linux/CMakeLists.txt. | ||
* See the documentation in windows/CMakeLists.txt. | ||
## Binding to native code | ||
```dart | ||
OrtEnv.instance.init(); | ||
``` | ||
|
||
To use the native code, bindings in Dart are needed. | ||
To avoid writing these by hand, they are generated from the header file | ||
(`src/onnxruntime.h`) by `package:ffigen`. | ||
Regenerate the bindings by running `flutter pub run ffigen --config ffigen.yaml`. | ||
### Creating the Session | ||
|
||
## Invoking native code | ||
```dart | ||
final sessionOptions = OrtSessionOptions(); | ||
const assetFileName = 'assets/models/test.onnx'; | ||
final rawAssetFile = await rootBundle.load(assetFileName); | ||
final bytes = rawAssetFile.buffer.asUint8List(); | ||
final session = OrtSession.fromBuffer(bytes, sessionOptions!); | ||
``` | ||
|
||
Very short-running native functions can be directly invoked from any isolate. | ||
For example, see `sum` in `lib/onnxruntime.dart`. | ||
### Performing inference | ||
|
||
Longer-running functions should be invoked on a helper isolate to avoid | ||
dropping frames in Flutter applications. | ||
For example, see `sumAsync` in `lib/onnxruntime.dart`. | ||
```dart | ||
final shape = [1, 2, 3]; | ||
final inputOrt = OrtValueTensor.createTensorWithDataList(data, shape); | ||
final inputs = {'input': inputOrt}; | ||
final runOptions = OrtRunOptions(); | ||
final outputs = await _session?.runAsync(runOptions, inputs); | ||
inputOrt.release(); | ||
runOptions.release(); | ||
``` | ||
|
||
## Flutter help | ||
### Releasing environment | ||
|
||
For help getting started with Flutter, view our | ||
[online documentation](https://flutter.dev/docs), which offers tutorials, | ||
samples, guidance on mobile development, and a full API reference. | ||
```dart | ||
OrtEnv.instance.release(); | ||
``` | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters