[ui] Fix endless log loading

[hellendag]

Summary & Motivation

We received a bug report from an org with some very, very long log output on some of their runs. Specifically, hundreds of thousands of large logs for individual runs. I noticed two issues during testing:

• When a chunk of logs loads and indicates that there are more older logs left to stream, we consider that a "loading" state and simply don't show any logs in the logs table at all.
  • This is unnecessary, since we're streaming logs in. Instead, just show the loading state if there are no logs available yet at all.
• First, we're using the spread-to-concat antipattern in the subscription handler reducer, which means iterating over the entire existing list of nodes before adding the newly received logs.
  • To resolve this, I'm changing the state array to track chunks of logs instead of flattened logs. This means we can more efficiently push chunks into state without mutating state or taking the time to copy entire chunks, then flatten them as needed for filtering/rendering.
  • I used [].concat(...chunks) as the flattening approach, based on benchmarking. it seems to be faster than flat().

How I Tested These Changes

View run with neverending log loading. Verify that logs appear and stream in.

Use Chrome profiler to determine that the chunking and flattening behavior appears to be performant at scale. The filterLogs function remains expensive, but I think in this case it's because the logs themselves are enormous and require a ton of JSON stringification to support text searching.

Changelog

[ui] Fix run log streaming for runs with a large volume of logs.

[hellendag]

• [ui] Fix endless log loading #26446 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Deploy preview for dagit-core-storybook ready!

✅ Preview
https://dagit-core-storybook-j1hezvw9k-elementl.vercel.app
https://dish-logs-loading-forever.core-storybook.dagster-docs.io

Built with commit c174946.
This pull request is being automatically deployed with vercel-action

[hellendag]

This fixes the specific bug report.

[hellendag]

Tweaking the loading state to better match our loading states elsewhere.

[hellendag]

Avoid copying the entire list of individual logs by using chunks instead.

[salazarm]

Could try using useThrottledMemo in case the websocket gets spammed very quickly then we can batch multiple updates to lead to just 1 flattenNodeChunks call.

[salazarm]

import React, { useState, useEffect } from 'react';

function withThrottle(WrappedComponent, delay = 1000) {
  return function ThrottledComponent(props) {
    const [throttledProps, setThrottledProps] = useState(props);

    useEffect(() => {
      const handler = setTimeout(() => setThrottledProps(props), delay);
      return () => clearTimeout(handler);
    }, [props, delay]);

    return <WrappedComponent {...throttledProps} />;
  };
}

export default withThrottle;

Or maybe we could use something like the above to throttle the number of re-renders for the logs component.

[hellendag]

There is throttling behavior in LogsProvider to guard against this. It looks like it is working as intended, though it's pretty old code and could maybe be revisited at some point.

[salazarm]

Yeah I added that throttling. Thinking back on it though I think throttling at the component level would be nicer than just throttling the websocket updates.

[hellendag]

Merge activity

• Dec 16, 1:35 PM CST: A user started a stack merge that includes this pull request via Graphite.
• Dec 16, 1:35 PM CST: A user merged this pull request with Graphite.