You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
the request pool is too small. For speculative decoding with llama3.1-8B, when the tree size is larger than 128, it has an error:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/jinweiy/deft/project_deft/DeFT/deft/tree_decoding/branch_controller.py", line 23, in apply_branching
return self.branching_function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/jinweiy/deft/project_deft/DeFT/deft/tree_decoding/generation/branch_func_example.py", line 403, in example_branch_Func4_SpeculativeDecoding
model.tree.branch(model.tree.root, token_tree_size)
File "/data/jinweiy/deft/project_deft/DeFT/deft/tree_decoding/tree_cache.py", line 361, in branch
assert new_req is not None
^^^^^^^^^^^^^^^^^^^
AssertionError
Steps To Reproduce
run speculative decoding with llama3.1-8B and set the tree size>=128.
Additional Information
No response
The text was updated successfully, but these errors were encountered:
Description of the bug
Two errors:
Steps To Reproduce
run speculative decoding with llama3.1-8B and set the tree size>=128.
Additional Information
No response
The text was updated successfully, but these errors were encountered: