Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Task exception was never retrieved #121

Open
ecsplendid opened this issue Dec 28, 2024 · 2 comments
Open

[BUG] Task exception was never retrieved #121

ecsplendid opened this issue Dec 28, 2024 · 2 comments
Labels
bug Something isn't working
Milestone
epic

Comments

@ecsplendid
Copy link

ecsplendid commented Dec 28, 2024 •
edited
Loading

Describe the bug

I have a pretty heavy xoscar app, over 100 actors running. I get this error randomly, perhaps about every 20-30 runs. Been happening for many months but thought I would report it here if the repo isn't dead.

socket.send() raised exception.                                                                                                   
Task exception was never retrieved
future: <Task finished name='Task-27' coro=<AbstractActorPool.process_message() done, defined at /Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py:410> exception=ConnectionResetError('Connection lost')>
Traceback (most recent call last):
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 426, in process_message
    await self._send_channel(processor.result, channel)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 378, in _send_channel
    await channel.send(result)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/communication/socket.py", line 88, in send
    await self.writer.drain()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 392, in drain
    await self._protocol._drain_helper()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
socket.send() raised exception.
Task exception was never retrieved
future: <Task finished name='Task-34' coro=<AbstractActorPool.process_message() done, defined at /Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py:410> exception=ConnectionResetError('Connection lost')>
Traceback (most recent call last):
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 426, in process_message
    await self._send_channel(processor.result, channel)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 378, in _send_channel
    await channel.send(result)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/communication/socket.py", line 88, in send
    await self.writer.drain()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 392, in drain
    await self._protocol._drain_helper()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost

To Reproduce

To help us to reproduce this bug, please provide information below:

  1. Your Python version - 3.11.10 (main, Oct 3 2024, 02:26:51) [Clang 14.0.6 ]
  2. The version of Xoscar you use - 0.4.0
  3. Versions of crucial packages, such as numpy, scipy and pandas

NumPy: version 1.26.4
• SciPy: version 1.14.1
• Pandas: version 2.2.3
Matplotlib: version 3.9.2
• Scikit-learn: version 1.5.2
• Pytest: version 8.3.3

I've just noticed im behind on versions, I wasn't aware this was still being updated, ive updated to 0.4.6 and will report back if fixed

  1. Full stack of the error.
  2. Minimized code to reproduce the error.

My code it too complex to give a repo but to give you some idea here is my pool setup

async def setup_actor_system(run_id: str):
    from markdown_actor import MarkdownGeneratorActor  # Import here to avoid circular import
    start_time = datetime.now()
    pool = None
    logging_db_actor = None
    try:
        # Clean up any existing processes first
        await cleanup_existing_processes()
        
        pool = await xo.create_actor_pool(
            address=ACTOR_POOL_ADDRESS,
            n_process=NUM_PROCESSES,
            subprocess_start_method='spawn'
        )

        # Create ActorRegistry
        registry = await xo.create_actor(ActorRegistry, address=pool.external_address, uid='actor_registry')

        # Create LoggingDatabaseActor first
        logging_db_actor = await xo.create_actor(
            LoggingDatabaseActor,
            address=pool.external_address,
            uid='LoggingDatabaseActor'
        )
        await registry.register_actor(actor_type='LoggingDatabaseActor', actor_ref=logging_db_actor)

        # Now we can log the initialization
        await logging_db_actor.info(
            run_id=run_id,
            actor_name="actor_pool",
            message="RUNTIME_LOG: Starting actor system initialization",
            json_data={"start_time": start_time.isoformat()}
        )

        steering_actor = await xo.create_actor(
            SteeringActor,
            address=pool.external_address,
            uid='SteeringActor'
        )

        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Actor pool created successfully", json_data=None)
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Actor registry created successfully", json_data=None)
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="LoggingDatabaseActor created successfully", json_data=None)

        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating GoogleSearchManagerActor...", json_data=None)
        google_search_manager = await xo.create_actor(
            GoogleSearchManagerActor,
            address=pool.external_address,
            uid='GoogleSearchManagerActor'
        )
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="GoogleSearchManagerActor created successfully", json_data=None)

        await registry.register_actor(actor_type='GoogleSearchManagerActor', actor_ref=google_search_manager)
        
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating DeepgramActor...", json_data=None)
        deepgram_actor = await xo.create_actor(DeepgramActor, address=pool.external_address, uid='DeepgramActor')
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="DeepgramActor created successfully", json_data=None)
        
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating DatabaseActor...", json_data=None)
        try:
            db_actor = await xo.create_actor(DatabaseActor, address=pool.external_address, uid='DatabaseActor')
            await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="DatabaseActor created successfully", json_data=None)
        except Exception as e:
            await logging_db_actor.error(run_id=run_id, actor_name="setup_actor_system", message=f"Error creating DatabaseActor: {str(e)}", json_data=None)
            raise
        
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating AudioPlayerActor...", json_data=None)
        audio_player_actor = await xo.create_actor(AudioPlayerActor, address=pool.external_address, uid='AudioPlayerActor')
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="AudioPlayerActor created successfully", json_data=None)
        
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating TranscriberActor...", json_data=None)
        transcriber_actor = await xo.create_actor(TranscriberActor, address=pool.external_address, uid='TranscriberActor')
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="TranscriberActor created successfully", json_data=None)
        
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating FragmentActor...", json_data=None)
        fragment_actor = await xo.create_actor(
            FragmentActor,
            db_actor_address=pool.external_address,
            address=pool.external_address,
            uid='FragmentActor'
        )
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="FragmentActor created successfully", json_data=None)
        
        anthropic_actors = []
        anthropic_actor_uids = []
        for i in range(NUM_AUDIOANALYSIS_ACTORS):
            uid = f'AnthropicActor_{i}'
            anthropic_actor = await xo.create_actor(AnthropicActor, 
                                                    address=pool.external_address, 
                                                    uid=uid, 
                                                    google_search_manager_address=pool.external_address,
                                                    run_id=run_id)
            anthropic_actors.append(anthropic_actor)
            anthropic_actor_uids.append(uid)
            await registry.register_actor(actor_type='AnthropicActor', actor_ref=anthropic_actor)
            await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message=f"Created {uid}", json_data=None)

        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating S3Actor...", json_data=None)
        s3_actor = await xo.create_actor(S3Actor, address=pool.external_address, uid='S3Actor')
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="S3Actor created successfully", json_data=None)
        
        # First create RateLimitManagerActor
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating RateLimitManagerActor...", json_data=None)
        rate_limit_manager = await xo.create_actor(
            RateLimitManagerActor,
            address=pool.external_address,
            uid='RateLimitManagerActor'
        )
        await registry.register_actor(actor_type='RateLimitManagerActor', actor_ref=rate_limit_manager)
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="RateLimitManagerActor created successfully", json_data=None)

        # Then create GeminiManagerActor
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating GeminiManagerActor...", json_data=None)
        gemini_manager = await xo.create_actor(
            GeminiManagerActor,
            address=pool.external_address,
            uid='GeminiManagerActor'
        )
        await registry.register_actor(actor_type='GeminiManagerActor', actor_ref=gemini_manager)
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="GeminiManagerActor created successfully", json_data=None)

        # Then create SummaryActor
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating SummaryActor...", json_data=None)
        summary_actor = await xo.create_actor(
            SummaryActor,
            anthropic_actor_address=pool.external_address,
            db_actor_address=pool.external_address,
            address=pool.external_address,
            uid='SummaryActor'
        )
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="SummaryActor created successfully", json_data=None)
        
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating SummaryManagerActor...", json_data=None)
        summary_manager = await xo.create_actor(
            SummaryManagerActor,
            address=pool.external_address,
            uid='SummaryManagerActor'
        )
        await registry.register_actor(actor_type='SummaryManagerActor', actor_ref=summary_manager)
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="SummaryManagerActor created successfully", json_data=None)
        
        # Create AudioAnalysisManagerActor
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating AudioAnalysisManagerActor...", json_data=None)
        audio_analysis_manager = await xo.create_actor(
            AudioAnalysisManagerActor,
            address=pool.external_address,
            uid='AudioAnalysisManagerActor',
            logging_db_actor=logging_db_actor
        )
        await registry.register_actor(actor_type='AudioAnalysisManagerActor', actor_ref=audio_analysis_manager)
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="AudioAnalysisManagerActor created successfully", json_data=None)

        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating AudioAnalysisRunnerActor...", json_data=None)
        audio_analysis_runner = await xo.create_actor(
            AudioAnalysisRunnerActor,
            address=pool.external_address,
            uid='AudioAnalysisRunnerActor'
        )
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="AudioAnalysisRunnerActor created successfully", json_data=None)

        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating SpeechmaticsActor...", json_data=None)
        speechmatics_actor = await xo.create_actor(
            SpeechmaticsActor,
            address=pool.external_address,
            uid='SpeechmaticsActor'
        )
        await registry.register_actor(actor_type='SpeechmaticsActor', actor_ref=speechmatics_actor)
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="SpeechmaticsActor created successfully", json_data=None)

        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating SpeechmaticsManagerActor...")
        speechmatics_manager = await xo.create_actor(
            SpeechmaticsManagerActor,
            address=pool.external_address,
            uid='SpeechmaticsManagerActor'
        )
        await registry.register_actor(actor_type='SpeechmaticsManagerActor', actor_ref=speechmatics_manager)
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="SpeechmaticsManagerActor created successfully")

        # Create TranscriptionRefinementManagerActor instances
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message=f"Creating {NUM_TRANSCRIPTION_REFINEMENT_MANAGERS} TranscriptionRefinementManagerActor instances...")
        refinement_managers = []
        for i in range(NUM_TRANSCRIPTION_REFINEMENT_MANAGERS):
            manager = await xo.create_actor(
                TranscriptionRefinementManagerActor,
                manager_id=i,
                anthropic_actor_address=pool.external_address,
                gemini_manager_address=pool.external_address,
                db_actor_address=pool.external_address,
                address=pool.external_address,
                uid=f'TranscriptionRefinementManagerActor_{i}'
            )
            refinement_managers.append(manager)
            await registry.register_actor(actor_type='TranscriptionRefinementManagerActor', actor_ref=manager)
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message=f"Created {NUM_TRANSCRIPTION_REFINEMENT_MANAGERS} TranscriptionRefinementManagerActor instances")

        # Create AudioExtractionActor instances and manager
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message=f"Creating AudioExtractionManagerActor and workers...")
        extraction_actors = []
        for i in range(NUM_AUDIO_EXTRACTION_WORKERS):
            actor = await xo.create_actor(
                AudioExtractionActor,
                address=pool.external_address,
                uid=f'AudioExtractionActor_{i}'
            )
            extraction_actors.append(actor)
            await registry.register_actor(actor_type='AudioExtractionActor', actor_ref=actor)
            await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message=f"Created AudioExtractionActor_{i}")

        extraction_manager = await xo.create_actor(
            AudioExtractionManagerActor,
            address=pool.external_address,
            uid='AudioExtractionManagerActor'
        )
        await registry.register_actor(actor_type='AudioExtractionManagerActor', actor_ref=extraction_manager)
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="AudioExtractionManagerActor created successfully")

        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating FragmentManagerActor...", json_data=None)
        fragment_manager_actor = await xo.create_actor(
            FragmentManagerActor,
            run_id=run_id,
            address=pool.external_address,
            uid='FragmentManagerActor'
        )
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="FragmentManagerActor created successfully", json_data=None)
        
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating TOCGeneratorActor...", json_data=None)
        toc_generator_actor = await xo.create_actor(
            TOCGeneratorActor,
            db_actor_address=pool.external_address,
            anthropic_actor_uids=anthropic_actor_uids,
            address=pool.external_address,
            uid='TOCGeneratorActor'
        )
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="TOCGeneratorActor created successfully", json_data=None)
        
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating SRTGeneratorActor...", json_data=None)
        srt_generator_actor = await xo.create_actor(
            SRTGeneratorActor,
            db_actor_address=pool.external_address,
            anthropic_actor_uids=anthropic_actor_uids,
            address=pool.external_address,
            uid='SRTGeneratorActor'
        )
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="SRTGeneratorActor created successfully", json_data=None)
        
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="Creating MarkdownGeneratorActor...")
        markdown_generator = await xo.create_actor(
            MarkdownGeneratorActor,
            db_actor_address=pool.external_address,
            address=pool.external_address,
            uid='MarkdownGeneratorActor'
        )
        await registry.register_actor(actor_type='MarkdownGeneratorActor', actor_ref=markdown_generator)
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message="MarkdownGeneratorActor created successfully")

        await registry.register_actor(actor_type='GeminiManagerActor', actor_ref=gemini_manager)
        await registry.register_actor(actor_type='AudioAnalysisManagerActor', actor_ref=audio_analysis_manager)
        await registry.register_actor(actor_type='AudioAnalysisRunnerActor', actor_ref=audio_analysis_runner)
        await registry.register_actor(actor_type='FragmentManagerActor', actor_ref=fragment_manager_actor)
        await registry.register_actor(actor_type='DeepgramActor', actor_ref=deepgram_actor)
        await registry.register_actor(actor_type='DatabaseActor', actor_ref=db_actor)
        await registry.register_actor(actor_type='AudioPlayerActor', actor_ref=audio_player_actor)
        await registry.register_actor(actor_type='TranscriberActor', actor_ref=transcriber_actor)
        await registry.register_actor(actor_type='FragmentActor', actor_ref=fragment_actor)
        await registry.register_actor(actor_type='S3Actor', actor_ref=s3_actor)
        await registry.register_actor(actor_type='SummaryActor', actor_ref=summary_actor)
        await registry.register_actor(actor_type='SummaryManagerActor', actor_ref=summary_manager)
        await registry.register_actor(actor_type='TOCGeneratorActor', actor_ref=toc_generator_actor)
        await registry.register_actor(actor_type='SRTGeneratorActor', actor_ref=srt_generator_actor)
        
        await logging_db_actor.info(run_id=run_id, actor_name="setup_actor_system", message=f"Created {len(anthropic_actors)} AnthropicActor instances", json_data=None)

        exec_summary_actor = await xo.create_actor(
            ExecSummaryActor,
            address=pool.external_address,
            uid='ExecSummaryActor'
        )
        await registry.register_actor(actor_type='ExecSummaryActor', actor_ref=exec_summary_actor)

        references_improvement_actor = await xo.create_actor(
            ReferencesImprovementActor,
            anthropic_actor_address=pool.external_address,
            db_actor_address=pool.external_address,
            google_search_actor_address=pool.external_address,
            address=pool.external_address,
            uid='ReferencesImprovementActor'
        )
        await registry.register_actor(actor_type='ReferencesImprovementActor', actor_ref=references_improvement_actor)

        end_time = datetime.now()
        runtime_seconds = (end_time - start_time).total_seconds()
        await logging_db_actor.info(
            run_id=run_id,
            actor_name="actor_pool",
            message="RUNTIME_LOG: Actor system initialization completed",
            json_data={
                "runtime_seconds": runtime_seconds,
                "start_time": start_time.isoformat(),
                "end_time": end_time.isoformat(),
                "num_actors_initialized": {
                    "anthropic_actors": len(anthropic_actors),
                    "refinement_managers": len(refinement_managers),
                    "extraction_actors": len(extraction_actors)
                }
            }
        )

        return (
            pool, registry, deepgram_actor, transcriber_actor, audio_player_actor, 
            fragment_actor, db_actor, anthropic_actors, s3_actor, fragment_manager_actor,
            summary_actor, toc_generator_actor, srt_generator_actor, audio_analysis_manager,
            audio_analysis_runner, gemini_manager, google_search_manager, speechmatics_actor,
            speechmatics_manager, logging_db_actor, exec_summary_actor, steering_actor,
            references_improvement_actor, refinement_managers, extraction_manager, extraction_actors,
            markdown_generator
        )
    except Exception as e:
        # Log initialization failure
        if logging_db_actor:
            await logging_db_actor.error(
                run_id=run_id,
                actor_name="actor_pool",
                message="RUNTIME_LOG: Actor system initialization failed",
                json_data={
                    "error": str(e),
                    "runtime_seconds": (datetime.now() - start_time).total_seconds()
                }
            )
        if pool:
            try:
                await pool.stop()
            except:
                pass  # Ignore errors during cleanup
        raise
@XprobeBot XprobeBot added the bug Something isn't working label Dec 28, 2024
@XprobeBot XprobeBot added this to the epic milestone Dec 28, 2024
@ecsplendid
Copy link
Author 

Corrupt value: 0x56311111d22f64f3
python(30230,0x32b01f000) malloc: *** set a breakpoint in malloc_error_break to debug

New error not seen before on 0.4.6

@ecsplendid
Copy link
Author

Got a slightly better error now after reverting to 0.4.3

zsh: segmentation fault  python ...
socket.send() raised exception.                                                                                                   
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
Task exception was never retrieved
future: <Task finished name='Task-26' coro=<AbstractActorPool.process_message() done, defined at /Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py:410> exception=ConnectionResetError('Connection lost')>
Traceback (most recent call last):
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 426, in process_message
    await self._send_channel(processor.result, channel)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 378, in _send_channel
    await channel.send(result)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/communication/socket.py", line 88, in send
    await self.writer.drain()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 392, in drain
    await self._protocol._drain_helper()
  File "/Users/timTask exception was never retrieved
future: <Task finished name='Task-22' coro=<AbstractActorPool.process_message() done, defined at /Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py:410> exception=ConnectionResetError('Connection lost')>
Traceback (most recent call last):
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 426, in process_message
    await self._send_channel(processor.result, channel)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 378, in _send_channel
    await channel.send(result)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/communication/socket.py", line 88, in send
    await self.writer.drain()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 392, in drain
    await self._protocol._drain_helper()
  File "/Users/timTask exception was never retrieved
future: <Task finished name='Task-22' coro=<AbstractActorPool.process_message() done, defined at /Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py:410> exception=ConnectionResetError('Connection lost')>
Traceback (most recent call last):
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 426, in process_message
    await self._send_channel(processor.result, channel)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 378, in _send_channel
    await channel.send(result)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/communication/socket.py", line 88, in send
    await self.writer.drain()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 392, in drain
    await self._protocol._drain_helper()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
scarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
scarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
Task exception was never retrieved
future: <Task finished name='Task-22' coro=<AbstractActorPool.process_message() done, defined at /Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py:410> exception=ConnectionResetError('Connection lost')>
Traceback (most recent call last):
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 426, in process_message
    await self._send_channel(processor.result, channel)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 378, in _send_channel
    await channel.send(result)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/communication/socket.py", line 88, in send
    await self.writer.drain()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 392, in drain
    await self._protocol._drain_helper()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
Task exception was never retrieved
future: <Task finished name='Task-23' coro=<AbstractActorPool.process_message() done, defined at /Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py:410> exception=ConnectionResetError('Connection lost')>
Traceback (most recent call last):
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 426, in process_message
    await self._send_channel(processor.result, channel)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 378, in _send_channel
    await channel.send(result)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/communication/socket.py", line 88, in send
    await self.writer.drain()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 392, in drain
    await self._protocol._drain_helper()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
Task exception was never retrieved
future: <Task finished name='Task-23' coro=<AbstractActorPool.process_message() done, defined at /Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py:410> exception=ConnectionResetError('Connection lost')>
Traceback (most recent call last):
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 426, in process_message
    await self._send_channel(processor.result, channel)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 378, in _send_channel
    await channel.send(result)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/communication/socket.py", line 88, in send
    await self.writer.drain()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 392, in drain
    await self._protocol._drain_helper()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
Task exception was never retrieved
future: <Task finished name='Task-28' coro=<AbstractActorPool.process_message() done, defined at /Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py:410> exception=ConnectionResetError('Connection lost')>
Traceback (most recent call last):
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 426, in process_message
    await self._send_channel(processor.result, channel)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 378, in _send_channel
    await channel.send(result)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/communication/socket.py", line 88, in send
    await self.writer.drain()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 392, in drain
    await self._protocol._drain_helper()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
Task exception was never retrieved
future: <Task finished name='Task-29' coro=<AbstractActorPool.process_message() done, defined at /Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py:410> exception=ConnectionResetError('Connection lost')>
Traceback (most recent call last):
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 426, in process_message
    await self._send_channel(processor.result, channel)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/pool.py", line 378, in _send_channel
    await channel.send(result)
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/site-packages/xoscar/backends/communication/socket.py", line 88, in send
    await self.writer.drain()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 392, in drain
    await self._protocol._drain_helper()
  File "/Users/timscarfe/miniconda3/envs/xoscar_env/lib/python3.11/asyncio/streams.py", line 166, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost

For some reason its now dying every time, not seen this before

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
epic
Development

No branches or pull requests
2 participants
@ecsplendid @XprobeBot