Skip to content
This repository has been archived by the owner on Sep 13, 2023. It is now read-only.

Support scipy sparse matrices #540

Closed
aguschin opened this issue Dec 23, 2022 · 1 comment
Closed

Support scipy sparse matrices #540

aguschin opened this issue Dec 23, 2022 · 1 comment
Labels
data-format Data format support p1-high High priority plugins Plugins and extensions for MLEM!

Comments

@aguschin
Copy link
Contributor

aguschin commented Dec 23, 2022

Support https://docs.scipy.org/doc//scipy/reference/sparse.html matrices/arrays as data formats for MLEM

Required to add MLEM to https://github.com/iterative/example-get-started

To reproduce, run dvc repro for the branch from aguschin/example-get-started#1

Traceback (most recent call last):
  File "/Users/aguschin/Git/iterative/example-get-started-gto/src/train.py", line 47, in <module>
    save(
  File "/Users/aguschin/.local/share/virtualenvs/gto-mP9zWfJO/lib/python3.9/site-packages/mlem/telemetry.py", line 50, in inner
    return f(*args, **kwargs)
  File "/Users/aguschin/.local/share/virtualenvs/gto-mP9zWfJO/lib/python3.9/site-packages/mlem/core/metadata.py", line 120, in save
    meta = get_object_metadata(
  File "/Users/aguschin/.local/share/virtualenvs/gto-mP9zWfJO/lib/python3.9/site-packages/mlem/core/metadata.py", line 53, in get_object_metadata
    return MlemModel.from_obj(
  File "/Users/aguschin/.local/share/virtualenvs/gto-mP9zWfJO/lib/python3.9/site-packages/mlem/core/objects.py", line 758, in from_obj
    sample_data, methods_sample_data = mlem_model._create_preprocessors(
  File "/Users/aguschin/.local/share/virtualenvs/gto-mP9zWfJO/lib/python3.9/site-packages/mlem/core/objects.py", line 687, in _create_preprocessors
    pre, pre_method = self._create_processor(
  File "/Users/aguschin/.local/share/virtualenvs/gto-mP9zWfJO/lib/python3.9/site-packages/mlem/core/objects.py", line 655, in _create_processor
    model_type = ModelAnalyzer.analyze(obj, sample_data=sample_data).bind(
  File "/Users/aguschin/.local/share/virtualenvs/gto-mP9zWfJO/lib/python3.9/site-packages/mlem/core/model.py", line 358, in analyze
    return super().analyze(
  File "/Users/aguschin/.local/share/virtualenvs/gto-mP9zWfJO/lib/python3.9/site-packages/mlem/core/hooks.py", line 107, in analyze
    return cls.find_hook(obj).process(obj, **kwargs)
  File "/Users/aguschin/.local/share/virtualenvs/gto-mP9zWfJO/lib/python3.9/site-packages/mlem/contrib/callable.py", line 213, in process
    s = Signature.from_method(
  File "/Users/aguschin/.local/share/virtualenvs/gto-mP9zWfJO/lib/python3.9/site-packages/mlem/core/model.py", line 212, in from_method
    returns = DataAnalyzer.analyze(result)
  File "/Users/aguschin/.local/share/virtualenvs/gto-mP9zWfJO/lib/python3.9/site-packages/mlem/core/hooks.py", line 107, in analyze
    return cls.find_hook(obj).process(obj, **kwargs)
  File "/Users/aguschin/.local/share/virtualenvs/gto-mP9zWfJO/lib/python3.9/site-packages/mlem/core/hooks.py", line 135, in find_hook
    raise HookNotFound(
mlem.core.errors.HookNotFound: No suitable DataHook for object of type "csr_matrix". Registered hooks: [<class 'mlem.core.data_type.PrimitiveType'>, <class 'mlem.core.data_type.OrderedCollectionHook'>, <class 'mlem.core.data_type.DictTypeHook'>, <class 'mlem.core.data_type.BinaryDataType'>, <class 'mlem.contrib.numpy.NumpyNumberType'>, <class 'mlem.contrib.numpy.NumpyNdarrayType'>]

under #423

@aguschin aguschin added plugins Plugins and extensions for MLEM! p1-high High priority data-format Data format support labels Dec 23, 2022
@aguschin aguschin changed the title Support numpy sparse matrices Support scipy sparse matrices Dec 23, 2022
@aguschin
Copy link
Contributor Author

Ok, so this one was actually fixed in #538. MLEM still doesn't support other formats except for CSR, but that's fine for now. We can add them later when needed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
data-format Data format support p1-high High priority plugins Plugins and extensions for MLEM!
Projects
None yet
Development

No branches or pull requests

1 participant