Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

:onyx/max-peers is not respected #35

Open
kamituel opened this issue May 15, 2018 · 1 comment
Open

:onyx/max-peers is not respected #35

kamituel opened this issue May 15, 2018 · 1 comment

Comments

@kamituel
Copy link

If :onyx/max-peers is set for the partition table task, but :onyx/n-peers is not, the task will break with the NullPointerException that is rather hard to track since it happens within the function passed to vreset!.

The docs in the README.md should either state clearly :onyx/n-peers is a required parameter for this task, or onyx.plugin.sql/partition-table should fall back to using :onyx/max-peers if :onyx/n-peers is nil. This fix could be applied here: https://github.com/onyx-platform/onyx-sql/blob/0.13.x/src/onyx/plugin/sql.clj#L70.

E.g. this:

;; Partition up the partitions over all n-peers.
    (take-nth (:onyx/n-peers task-map)
              (drop slot-id
                    (map (fn [[l h]]
                           [l (dec (or h (inc n-max)))])
          ranges)))

Could become this:

;; Partition up the partitions over all n-peers.
    (take-nth (or (:onyx/n-peers task-map) (:onyx/max-peers task-map))
              (drop slot-id
                    (map (fn [[l h]]
                           [l (dec (or h (inc n-max)))])
          ranges)))
@lbradstreet
Copy link
Member

Given the way it partitions the data, I think the only real fix here is to throw when onyx/n-peers is not used. Given the partitioning scheme, I think the number of peers needs to be stable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants