Add support for Weave evaluation #264
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: ayulockin <[email protected]>
Signed-off-by: ayulockin <[email protected]>
Signed-off-by: ayulockin <[email protected]>
Signed-off-by: ayulockin <[email protected]>
mdemoret-nv
added
feature request
Signed-off-by: ayulockin <[email protected]>
Signed-off-by: ayulockin <[email protected]>
Signed-off-by: ayulockin <[email protected]>
Signed-off-by: ayulockin <[email protected]>
This is without weave turned on:
This is with weave enabled:
Note that this comparison is done only on 5 examples but definitely there is some performance overhead introduced when weave is turned on. |
ayulockin
commented
May 30, 2025
| @@ -16,6 +16,11 @@ | |||
|
|
|||
| general: | |||
| use_uvloop: true | |||
| telemetry: | |||
There was a problem hiding this comment.
TODO: remove this file later
Comment on lines
+11
to
+26
| def _unraisable_hook(unraisable): | ||
| # This is a workaround to avoid some weird concurrency issues | ||
| # the result of which is that we see ugly tracebacks on program exit. | ||
| # We are still investigating the root cause of this issue. | ||
| err = unraisable.exc_value | ||
| msg = str(err) | ||
| # drop both GeneratorExit–related and CancelScope–related noise | ||
| if isinstance(err, RuntimeError) and ( | ||
| "async generator ignored GeneratorExit" in msg | ||
| or "Attempted to exit cancel scope" in msg | ||
| or "unknown async library, or not in async context" in msg | ||
| ): | ||
| return | ||
| sys.__unraisablehook__(unraisable) | ||
|
|
||
| sys.unraisablehook = _unraisable_hook |
There was a problem hiding this comment.
If I run aiq eval --config_file=examples/simple/configs/eval_config_weave.yml by commenting out line 26, I get the following traceback:
stdout:
``` 2025-05-30 15:11:05,748 - aiq.eval.evaluate - INFO - Starting evaluation run with config file: examples/simple/configs/eval_config_weave.yml 2025-05-30 15:11:06,193 - aiq.runtime.loader - WARNING - Loading module 'aiq.agent.register' from entry point 'aiq_agents' took a long time (199.633837 ms). Ensure all imports are inside your registered functions. 2025-05-30 15:11:06,499 - aiq.eval.evaluate - INFO - Cleaning up output directory .tmp/aiq/examples/simple 2025-05-30 15:11:06,565 - gql.transport.requests - INFO - >>> {"query": "query DefaultEntity {\n viewer {\n username\n defaultEntity {\n name\n }\n }\n}"} 2025-05-30 15:11:06,978 - gql.transport.requests - INFO - <<< {"data":{"viewer":{"username":"ayut","defaultEntity":{"name":"ayut"}}}} 2025-05-30 15:11:10,862 - gql.transport.requests - INFO - >>> {"query": "query DefaultEntity {\n viewer {\n username\n defaultEntity {\n name\n }\n }\n}"} 2025-05-30 15:11:11,263 - gql.transport.requests - INFO - <<< {"data":{"viewer":{"username":"ayut","defaultEntity":{"name":"ayut"}}}} weave: Logged in as Weights & Biases user: ayut. weave: View Weave data at https://wandb.ai/ayut/aiqtoolkit-eval/weave 2025-05-30 15:11:12,114 - weave.trace.init_message - INFO - Logged in as Weights & Biases user: ayut. View Weave data at https://wandb.ai/ayut/aiqtoolkit-eval/weave 2025-05-30 15:11:12,138 - aiq.profiler.utils - WARNING - Discovered frameworks: {} in function webquery_tool by inspecting source. It is recommended and more reliable to instead add the used LLMFrameworkEnum types in the framework_wrappers argument when calling @register_function. 2025-05-30 15:11:12,149 - langchain_community.utils.user_agent - WARNING - USER_AGENT environment variable not set, consider setting it to identify your requests. 2025-05-30 15:11:12,163 - aiq_simple.register - INFO - Generating docs for the webpage: https://docs.smith.langchain.com Fetching pages: 100%|################################################################################| 1/1 [00:00<00:00, 3.68it/s] 2025-05-30 15:11:13,055 - faiss.loader - INFO - Loading faiss. 2025-05-30 15:11:13,092 - faiss.loader - INFO - Successfully loaded faiss. Running workflow: 0%| | 0/5 [00:00 to , which may lead to unintended conversions. Consider adding a direct converter from to to ensure correctness. Running workflow: 20%|███████████████▌ | 1/5 [00:04<00:16, 4.15s/it]Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 404, in __aiter__ yield part RuntimeError: async generator ignored GeneratorExit Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 348, in aclose await self._connection._response_closed() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 239, in _response_closed async with self._state_lock: ^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_synchronization.py", line 77, in __aenter__ await self._anyio_lock.acquire() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1799, in acquire await AsyncIOBackend.cancel_shielded_checkpoint() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2349, in cancel_shielded_checkpoint with CancelScope(shield=True): ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 457, in __exit__ raise RuntimeError( RuntimeError: Attempted to exit cancel scope in a different task than it was entered in Running workflow: 40%|███████████████████████████████▏ | 2/5 [00:04<00:05, 1.83s/it]Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 340, in __aiter__ with AsyncShieldCancellation(): ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_synchronization.py", line 203, in __init__ self._backend = current_async_library() ^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_synchronization.py", line 30, in current_async_library environment = sniffio.current_async_library() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/sniffio/_impl.py", line 93, in current_async_library raise AsyncLibraryNotFoundError( sniffio._impl.AsyncLibraryNotFoundError: unknown async library, or not in async context Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 406, in __aiter__ await self.aclose() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 412, in aclose with AsyncShieldCancellation(): ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_synchronization.py", line 203, in __init__ self._backend = current_async_library() ^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_synchronization.py", line 30, in current_async_library environment = sniffio.current_async_library() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/sniffio/_impl.py", line 93, in current_async_library raise AsyncLibraryNotFoundError( sniffio._impl.AsyncLibraryNotFoundError: unknown async library, or not in async context Running workflow: 100%|██████████████████████████████████████████████████████████████████████████████| 5/5 [00:06<00:00, 1.29s/it] Evaluating Ragas nv_accuracy: 0%| | 0/5 [00:00 | 1/5 [00:00<00:03, 1.11it/s] Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 404, in __aiter__ yield part RuntimeError: async generator ignored GeneratorExit Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 404, in __aiter__ yield part RuntimeError: async generator ignored GeneratorExit Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 404, in __aiter__ yield part RuntimeError: async generator ignored GeneratorExit Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 404, in __aiter__ yield part RuntimeError: async generator ignored GeneratorExit Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 404, in __aiter__ yield part RuntimeError: async generator ignored GeneratorExit Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 404, in __aiter__ yield part RuntimeError: async generator ignored GeneratorExit Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 404, in __aiter__ yield part RuntimeError: async generator ignored GeneratorExit Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 404, in __aiter__ yield part RuntimeError: async generator ignored GeneratorExit Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 348, in aclose 2025-05-30 15:11:22,044 - openai._base_client - INFO - Retrying request to /chat/completions in 0.441259 seconds await self._connection._response_closed() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 239, in _response_closed async with self._state_lock: ^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_synchronization.py", line 77, in __aenter__ await self._anyio_lock.acquire() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1799, in acquire await AsyncIOBackend.cancel_shielded_checkpoint() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2349, in cancel_shielded_checkpoint with CancelScope(shield=True): ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 456, in __exit__ if current_task() is not self._host_task: ^^^^^^^^^^^^^^ RuntimeError: no running event loop Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 348, in aclose await self._connection._response_closed() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 239, in _response_closed async with self._state_lock: ^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_synchronization.py", line 77, in __aenter__ await self._anyio_lock.acquire() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1799, in acquire await AsyncIOBackend.cancel_shielded_checkpoint() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2349, in cancel_shielded_checkpoint with CancelScope(shield=True): ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 456, in __exit__ if current_task() is not self._host_task: ^^^^^^^^^^^^^^ RuntimeError: no running event loop Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 348, in aclose await self._connection._response_closed() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 239, in _response_closed async with self._state_lock: ^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_synchronization.py", line 77, in __aenter__ await self._anyio_lock.acquire() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1799, in acquire await AsyncIOBackend.cancel_shielded_checkpoint() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2349, in cancel_shielded_checkpoint with CancelScope(shield=True): ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 456, in __exit__ if current_task() is not self._host_task: ^^^^^^^^^^^^^^ RuntimeError: no running event loop Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 348, in aclose await self._connection._response_closed() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 239, in _response_closed async with self._state_lock: ^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_synchronization.py", line 77, in __aenter__ await self._anyio_lock.acquire() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1799, in acquire await AsyncIOBackend.cancel_shielded_checkpoint() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2349, in cancel_shielded_checkpoint with CancelScope(shield=True): ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 456, in __exit__ if current_task() is not self._host_task: ^^^^^^^^^^^^^^ RuntimeError: no running event loop Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 348, in aclose await self._connection._response_closed() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 239, in _response_closed async with self._state_lock: ^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_synchronization.py", line 77, in __aenter__ await self._anyio_lock.acquire() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1799, in acquire await AsyncIOBackend.cancel_shielded_checkpoint() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2349, in cancel_shielded_checkpoint with CancelScope(shield=True): ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 456, in __exit__ if current_task() is not self._host_task: ^^^^^^^^^^^^^^ RuntimeError: no running event loop Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 348, in aclose await self._connection._response_closed() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 239, in _response_closed async with self._state_lock: ^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_synchronization.py", line 77, in __aenter__ await self._anyio_lock.acquire() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1799, in acquire await AsyncIOBackend.cancel_shielded_checkpoint() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2349, in cancel_shielded_checkpoint with CancelScope(shield=True): ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 456, in __exit__ if current_task() is not self._host_task: ^^^^^^^^^^^^^^ RuntimeError: no running event loop Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 348, in aclose await self._connection._response_closed() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 239, in _response_closed async with self._state_lock: ^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_synchronization.py", line 77, in __aenter__ await self._anyio_lock.acquire() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1799, in acquire await AsyncIOBackend.cancel_shielded_checkpoint() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2349, in cancel_shielded_checkpoint with CancelScope(shield=True): ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 456, in __exit__ if current_task() is not self._host_task: ^^^^^^^^^^^^^^ RuntimeError: no running event loop Exception ignored in: Traceback (most recent call last): File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 348, in aclose await self._connection._response_closed() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 239, in _response_closed async with self._state_lock: ^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/httpcore/_synchronization.py", line 77, in __aenter__ await self._anyio_lock.acquire() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1799, in acquire await AsyncIOBackend.cancel_shielded_checkpoint() File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2349, in cancel_shielded_checkpoint with CancelScope(shield=True): ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/ayushthakur/integrations/aiq/AIQToolkit/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 456, in __exit__ if current_task() is not self._host_task: ^^^^^^^^^^^^^^ RuntimeError: no running event loop Evaluating Ragas nv_context_relevance: 100%|█████████████████████████████████████████████████████████| 5/5 [00:03<00:00, 1.46it/s] Evaluating Ragas nv_response_groundedness: 100%|█████████████████████████████████████████████████████| 5/5 [00:04<00:00, 1.17it/s] Evaluating Ragas nv_accuracy: 100%|██████████████████████████████████████████████████████████████████| 5/5 [00:08<00:00, 1.77s/it] Evaluating Trajectory: 100%|█████████████████████████████████████████████████████████████████████████| 5/5 [00:07<00:00, 1.53s/it] 2025-05-30 15:11:29,512 - aiq.profiler.profile_runner - INFO - Wrote combined data to: .tmp/aiq/examples/simple/all_requests_profiler_traces.jsonjectory: 20%|██████████████▌ | 1/5 [00:05<00:21, 5.39s/it] 2025-05-30 15:11:29,578 - aiq.profiler.profile_runner - INFO - Wrote merged standardized DataFrame to .tmp/aiq/examples/simple/standardized_data_all.csv 2025-05-30 15:11:29,639 - aiq.profiler.profile_runner - INFO - Wrote inference optimization results to: .tmp/aiq/examples/simple/inference_optimization.json 2025-05-30 15:11:29,936 - aiq.profiler.profile_runner - INFO - Nested stack analysis complete 2025-05-30 15:11:29,954 - aiq.profiler.profile_runner - INFO - Concurrency spike analysis complete 2025-05-30 15:11:29,955 - aiq.profiler.profile_runner - INFO - Wrote workflow profiling report to: .tmp/aiq/examples/simple/workflow_profiling_report.txt 2025-05-30 15:11:29,956 - aiq.profiler.profile_runner - INFO - Wrote workflow profiling metrics to: .tmp/aiq/examples/simple/workflow_profiling_metrics.json 2025-05-30 15:11:29,959 - aiq.eval.evaluate - INFO - Workflow output written to .tmp/aiq/examples/simple/workflow_output.json 2025-05-30 15:11:29,960 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple/rag_relevance_output.json 2025-05-30 15:11:29,960 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple/rag_groundedness_output.json 2025-05-30 15:11:29,960 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple/rag_accuracy_output.json 2025-05-30 15:11:29,961 - aiq.eval.evaluate - INFO - Evaluation results written to .tmp/aiq/examples/simple/trajectory_accuracy_output.json 2025-05-30 15:11:29,961 - aiq.eval.utils.output_uploader - INFO - No S3 config provided; skipping upload. ```Right now I am using this hack to suppress tracebacks. These tracebacks are non-blocking i.e, the evaluation runs to completion and everything is correctly logged to Weave.
This integration took only a days work but I had spent a lot of time already to find root cause of the tracebacks I am seeing. Just thought of looping you all in.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds the ability to run evaluations such that the traces and evaluation scores are logged to W&B Weave. This allows for comparing evaluations, debugging evaluations, etc.
By Submitting this PR I confirm: