Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the Segment Anything Model to KerasCV #1987

Merged
merged 15 commits into from
Sep 19, 2023
Merged

Add the Segment Anything Model to KerasCV #1987

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
ddebecd
Start adding components for the segment anything model
tirthasheshpatel Jul 28, 2023
025dd15
SAMLayerNormalization -> keras.layers.LayerNormalization
tirthasheshpatel Jul 30, 2023
43b0f2b
Move the image encoder to detectron2 backbone and fix for tf.keras ba…
tirthasheshpatel Aug 2, 2023
ac7f30e
Address review comments and address saving bug
tirthasheshpatel Aug 18, 2023
360baf7
Make the backbone functional; unite MLP and MLPBlock
tirthasheshpatel Aug 21, 2023
6cb8c2f
Merge branch 'master' of github.com:keras-team/keras-cv into add-sam
tirthasheshpatel Aug 25, 2023
d7a1eb3
Address David's review comments
tirthasheshpatel Aug 25, 2023
ada6040
Add SAM Task model; make MaskDecoder and PromptEncoder XLA compatible
tirthasheshpatel Sep 8, 2023
93eb78f
Merge branch 'master' of github.com:keras-team/keras-cv into add-sam
tirthasheshpatel Sep 9, 2023
d908527
Remove a stray file
tirthasheshpatel Sep 9, 2023
07cb9d6
Add docs for the Task model
tirthasheshpatel Sep 10, 2023
343557a
Add more references
tirthasheshpatel Sep 11, 2023
19312e7
Remove SerializableSequential layer
tirthasheshpatel Sep 15, 2023
e5767d8
detectron2 -> vit_det; add SAM presets; fix ViTDet presets
tirthasheshpatel Sep 17, 2023
8ad94ae
Increse test tolerence for GCB Run
tirthasheshpatel Sep 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions keras_cv/layers/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@
from tensorflow.keras.layers import RandomWidth

from keras_cv.layers.augmenter import Augmenter
from keras_cv.layers.detectron2_layers import AddPositionalEmbedding
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to put these under a detectron2 namespace?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this would be exported as part of the public API - we have PatchingAndEmbedding which does patching with a Conv2D and then adds embeddings in this same form. Do we want to update that layer to use AddPositionalEmbedding as well for conformity?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't adding a new layer in a pre-existing class invalidate the weights set for the ViT model? Also, since PatchingAndEmbedding is still a TensorFlow Keras layer, I think, for the time being, it'd be easier to keep the two separate.

Although, I'd add a comment about this as a TODO so we don't forget to do it in the future. What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

from keras_cv.layers.detectron2_layers import MultiHeadAttentionWithRelativePE
from keras_cv.layers.detectron2_layers import ViTDetPatchingAndEmbedding
from keras_cv.layers.detectron2_layers import WindowedTransformerEncoder
from keras_cv.layers.feature_pyramid import FeaturePyramid
from keras_cv.layers.fusedmbconv import FusedMBConvBlock
from keras_cv.layers.mbconv import MBConvBlock
Expand Down Expand Up @@ -124,6 +128,7 @@
from keras_cv.layers.regularization.dropblock_2d import DropBlock2D
from keras_cv.layers.regularization.squeeze_excite import SqueezeAndExcite2D
from keras_cv.layers.regularization.stochastic_depth import StochasticDepth
from keras_cv.layers.serializable_sequential import SerializableSequential
from keras_cv.layers.spatial_pyramid import SpatialPyramidPooling
from keras_cv.layers.transformer_encoder import TransformerEncoder
from keras_cv.layers.vit_layers import PatchingAndEmbedding
Loading