perf: optimize kitty plot interactions

Add Kitty image-id prefetching for interactive plot frames so repeated pan and zoom can place already transmitted images instead of resending pixel payloads.\n\nKeep plot updates stable by tracking the previously visible image placement and deleting only that placement after the replacement has been drawn. Expand PTY perf coverage and architecture notes for the new prefetch and placement path.

Author: siriusctrl

docs/architecture.md

@@ -110,18 +110,36 @@ rasterization is still gated until an SVG rasterizer is added.
 Interactive plot viewing keeps terminal input ahead of expensive protocol
 payload work. The event loop drains pending key and resize events before drawing
 so burst input renders the latest state instead of every intermediate state. It
-also caches the last rendered frame by protocol, plot kind, viewport, and
-terminal size. Kitty frames request the full terminal cell area while rendering
-normal terminal windows at the full terminal pixel estimate. Very large windows
-use a bounded internal raster budget to keep redraw and protocol encoding cost
-predictable. Plot Kitty frames use zlib-compressed raw RGBA direct-data payloads
-so terminal updates avoid PNG decode work while still working when the terminal
-process cannot read files from the app's filesystem, such as SSH, container, or
-sandboxed sessions. The terminal chrome is rendered as a styled status bar with
-a stable dark background and segmented state text. For plot pixel protocols,
-the chrome also owns the header, legend, and axis labels so crisp terminal text
-surrounds a smaller body-only image payload; static chrome is repainted only on
-first draw or resize.
+keeps a bounded cache of encoded frames by protocol, plot kind, viewport, and
+terminal size. After user navigation, a small background prefetcher warms likely
+next frames without blocking the foreground draw. For repeated pan actions on
+large scenes, the prefetcher can render a transparent marks atlas once and crop
+future same-zoom pan frames, then composite those marks over the current
+grid/frame layer so axis labels and grid lines stay correct.
+
+Kitty plot frames use zlib-compressed raw RGBA direct-data payloads so terminal
+updates avoid PNG decode work while still working when the terminal process
+cannot read files from the app's filesystem, such as SSH, container, or
+sandboxed sessions. Prefetched Kitty frames are transmitted with image IDs
+during idle time. If the next key lands on an already-transmitted frame, the
+foreground path writes only a small image placement command instead of sending
+the image bytes again. Each visible plot image uses a stable placement id and a
+unique image id. Updates place the new image first, then delete only the
+previous visible image placement by image id. This keeps old pixels visible
+until replacement pixels are placed while avoiding broad z-index or full-screen
+deletes that can blank the plot during fast navigation. The prefetch list stays
+intentionally small: more candidates increase background raster work and hidden
+terminal bytes, so newer directional batches suppress stale, not-yet-transmitted
+candidates.
+
+Kitty frames request the full terminal cell area while rendering normal terminal
+windows at the full terminal pixel estimate. Very large windows use a bounded
+internal raster budget to keep redraw and protocol encoding cost predictable.
+The terminal chrome is rendered as a styled status bar with a stable dark
+background and segmented state text. For plot pixel protocols, the chrome also
+owns the header, legend, and axis labels so crisp terminal text surrounds a
+smaller body-only image payload; static chrome is repainted only on first draw
+or resize.
 
 ## Current Profiles
 

docs/testing.md

@@ -72,6 +72,14 @@ benchmark drives a direct PTY and avoids tmux passthrough ambiguity, but the
 millisecond timings still stop at PTY-observable bytes rather than at terminal
 GPU composition or physical display scanout.
 
+The `*_prefetched` metrics wait briefly between repeated navigation actions.
+They measure whether the direction-biased encoded-frame cache and pan prefetch
+path are actually helping repeated `+` and arrow-key interactions, separate
+from the uncached first keypress metrics. For Kitty, a healthy prefetched
+navigation hit should show `payload_bytes_delta` near zero because the
+foreground update is an image placement command for an idle-transmitted image,
+not a new image transfer.
+
 ## Selector Tests
 
 Selector tests live in `src/render/terminal.rs`.

scripts/bench-plot-e2e.sh

@@ -66,6 +66,10 @@ def payload_count(data):
     current = False
     for match in PAYLOAD_RE.finditer(data):
         controls = match.group(1).decode("ascii", "ignore")
+        if "a=p" in controls:
+            count += 1
+            current = False
+            continue
         if "a=T" in controls:
             if "t=f" in controls or "m=1" not in controls:
                 count += 1
@@ -236,6 +240,28 @@ def run_iteration(iteration):
             after_bytes - before_bytes,
         )
 
+        before_payloads = after_payloads
+        before_payload_bytes = after_payload_bytes
+        before_bytes = after_bytes
+        time.sleep(0.04)
+        read_available(session.master, session.output)
+        before_payloads = payload_count(session.output)
+        before_payload_bytes = payload_bytes(session.output)
+        before_bytes = len(session.output)
+        start = time.perf_counter()
+        session.send(b"+")
+        after_payloads, end = wait_for_payload_count(session.master, session.output, before_payloads + 1)
+        after_payload_bytes = payload_bytes(session.output)
+        after_bytes = len(session.output)
+        print_metric(
+            "plot_e2e_key_zoom_prefetched",
+            iteration,
+            int((end - start) * 1000),
+            after_payloads - before_payloads,
+            after_payload_bytes - before_payload_bytes,
+            after_bytes - before_bytes,
+        )
+
         before_payloads = after_payloads
         before_payload_bytes = after_payload_bytes
         before_bytes = after_bytes
@@ -253,6 +279,28 @@ def run_iteration(iteration):
             after_bytes - before_bytes,
         )
 
+        before_payloads = after_payloads
+        before_payload_bytes = after_payload_bytes
+        before_bytes = after_bytes
+        time.sleep(0.04)
+        read_available(session.master, session.output)
+        before_payloads = payload_count(session.output)
+        before_payload_bytes = payload_bytes(session.output)
+        before_bytes = len(session.output)
+        start = time.perf_counter()
+        session.send(b"\x1b[C")
+        after_payloads, end = wait_for_payload_count(session.master, session.output, before_payloads + 1)
+        after_payload_bytes = payload_bytes(session.output)
+        after_bytes = len(session.output)
+        print_metric(
+            "plot_e2e_key_pan_prefetched",
+            iteration,
+            int((end - start) * 1000),
+            after_payloads - before_payloads,
+            after_payload_bytes - before_payload_bytes,
+            after_bytes - before_bytes,
+        )
+
         before_payloads = after_payloads
         before_payload_bytes = after_payload_bytes
         before_bytes = after_bytes

src/render/protocols/kitty.rs

@@ -31,6 +31,63 @@ pub(crate) fn render_rgba_zlib_for_size(
     render_with_rgba_zlib_chunks_for_size(image, Some((columns.max(1), rows.max(1))))
 }
 
+pub(crate) fn render_rgba_zlib_for_size_with_id(
+    image: &RgbaImage,
+    image_id: u32,
+    columns: u32,
+    rows: u32,
+) -> Result<String> {
+    let z_index = z_index_for_image_id(image_id);
+    let compressed =
+        zlib_compress_bytes(image.as_raw()).context("compressing kitty RGBA payload")?;
+    let encoded = STANDARD.encode(compressed);
+    render_chunked_payload_str_with_action(
+        image.width(),
+        image.height(),
+        Some((columns.max(1), rows.max(1))),
+        &encoded,
+        "a=T",
+        &format!("i={image_id},p=1,z={z_index},f=32,o=z,q=2"),
+        false,
+    )
+}
+
+pub(crate) fn transmit_rgba_zlib_with_id(image: &RgbaImage, image_id: u32) -> Result<String> {
+    let compressed =
+        zlib_compress_bytes(image.as_raw()).context("compressing kitty RGBA payload")?;
+    let encoded = STANDARD.encode(compressed);
+    render_chunked_payload_str_with_action(
+        image.width(),
+        image.height(),
+        None,
+        &encoded,
+        "a=t",
+        &format!("i={image_id},f=32,o=z,q=2"),
+        false,
+    )
+}
+
+pub(crate) fn place_image_for_size(image_id: u32, columns: u32, rows: u32) -> String {
+    let z_index = z_index_for_image_id(image_id);
+    format!(
+        "{KITTY_PREFIX}a=p,i={image_id},p=1,z={z_index},c={},r={},C=1,q=2;{KITTY_SUFFIX}",
+        columns.max(1),
+        rows.max(1)
+    )
+}
+
+pub(crate) fn delete_visible_placements() -> &'static str {
+    "\u{1b}_Ga=d,q=2;\u{1b}\\"
+}
+
+fn z_index_for_image_id(image_id: u32) -> i32 {
+    if image_id % 2 == 0 { 0 } else { 1 }
+}
+
+pub(crate) fn delete_image_placement(image_id: u32, placement_id: u32) -> String {
+    format!("{KITTY_PREFIX}a=d,d=i,i={image_id},p={placement_id},q=2;{KITTY_SUFFIX}")
+}
+
 fn render_with_png_chunks(image: &DynamicImage) -> Result<String> {
     render_with_png_chunks_for_size(image, None)
 }
@@ -90,11 +147,15 @@ fn render_chunked_payload_with_format(
     format_control: &str,
 ) -> Result<String> {
     if chunks.is_empty() {
-        return Ok(format!("{KITTY_PREFIX}f=100,t=d,m=0;\u{1b}\\"));
+        return Ok(format!(
+            "{}{KITTY_PREFIX}f=100,t=d,m=0;\u{1b}\\",
+            delete_visible_placements()
+        ));
     }
 
     let payload_len = chunks.iter().map(String::len).sum::<usize>();
     let mut output = String::with_capacity(payload_len + chunks.len() * 64);
+    output.push_str(delete_visible_placements());
 
     for (index, chunk) in chunks.iter().enumerate() {
         let is_last = index + 1 == chunks.len();
@@ -124,13 +185,41 @@ fn render_chunked_payload_str_with_format(
     display_cells: Option<(u32, u32)>,
     base64_payload: &str,
     format_control: &str,
+) -> Result<String> {
+    render_chunked_payload_str_with_action(
+        width,
+        height,
+        display_cells,
+        base64_payload,
+        "a=T",
+        format_control,
+        true,
+    )
+}
+
+fn render_chunked_payload_str_with_action(
+    width: u32,
+    height: u32,
+    display_cells: Option<(u32, u32)>,
+    base64_payload: &str,
+    action_control: &str,
+    format_control: &str,
+    clear_before_display: bool,
 ) -> Result<String> {
     let chunks = chunked_base64_payload(base64_payload, 4096);
     if chunks.is_empty() {
-        return Ok(format!("{KITTY_PREFIX}f=100,t=d,m=0;\u{1b}\\"));
+        let clear = if clear_before_display {
+            delete_visible_placements()
+        } else {
+            ""
+        };
+        return Ok(format!("{clear}{KITTY_PREFIX}f=100,t=d,m=0;\u{1b}\\"));
     }
 
     let mut output = String::with_capacity(base64_payload.len() + chunks.len() * 64);
+    if clear_before_display {
+        output.push_str(delete_visible_placements());
+    }
     for (index, chunk) in chunks.iter().enumerate() {
         let is_last = index + 1 == chunks.len();
         let chunk_mode = if is_last { 0 } else { 1 };
@@ -141,7 +230,7 @@ fn render_chunked_payload_str_with_format(
                 .map(|(columns, rows)| format!(",c={columns},r={rows},C=1"))
                 .unwrap_or_default();
             output.push_str(&format!(
-                "a=T,{format_control},t=d,s={width},v={height}{display},m={chunk_mode};{chunk}",
+                "{action_control},{format_control},t=d,s={width},v={height}{display},m={chunk_mode};{chunk}",
                 chunk = chunk
             ));
         } else {
@@ -164,6 +253,7 @@ mod tests {
         let payload = super::render(&image).unwrap();
         assert!(payload.starts_with("\u{1b}_G"));
         assert!(payload.ends_with("\u{1b}\\"));
+        assert!(payload.starts_with(super::delete_visible_placements()));
         assert!(payload.contains("t=d"));
     }
 
@@ -176,6 +266,37 @@ mod tests {
         assert!(payload.contains(",c=80,r=24,C=1,"));
     }
 
+    #[test]
+    fn kitty_can_transmit_and_place_image_by_id() {
+        let image = ImageBuffer::from_pixel(2, 2, Rgba([0, 0, 0, 255]));
+
+        let transmit = super::transmit_rgba_zlib_with_id(&image, 42).unwrap();
+        let place = super::place_image_for_size(42, 80, 24);
+
+        assert!(transmit.contains("a=t,"));
+        assert!(transmit.contains("i=42"));
+        assert!(transmit.contains("f=32,o=z"));
+        assert!(transmit.contains("q=2"));
+        assert!(!transmit.contains(",c=80,r=24"));
+        assert!(!transmit.contains("a=d"));
+        assert_eq!(place, "\u{1b}_Ga=p,i=42,p=1,z=0,c=80,r=24,C=1,q=2;\u{1b}\\");
+    }
+
+    #[test]
+    fn kitty_id_display_places_without_predelete() {
+        let image = ImageBuffer::from_pixel(2, 2, Rgba([0, 0, 0, 255]));
+
+        let payload = super::render_rgba_zlib_for_size_with_id(&image, 7, 80, 24).unwrap();
+
+        assert!(!payload.starts_with(super::delete_visible_placements()));
+        assert!(payload.contains("a=T,i=7,p=1,z=1,f=32,o=z,q=2"));
+        assert!(!payload.contains("a=d"));
+        assert_eq!(
+            super::delete_image_placement(7, 1),
+            "\u{1b}_Ga=d,d=i,i=7,p=1,q=2;\u{1b}\\"
+        );
+    }
+
     #[test]
     fn kitty_multichunk_payload_uses_continuation_headers_only_after_first_chunk() {
         let image = image::DynamicImage::ImageRgba8(ImageBuffer::from_fn(96, 96, |x, y| {
@@ -192,8 +313,9 @@ mod tests {
             .collect::<Vec<_>>();
 
         assert!(packets.len() > 1);
-        assert!(packets[0].starts_with("a=T,f=100,t=d,"));
-        for packet in packets.iter().skip(1) {
+        assert!(packets[0].starts_with("a=d,q=2"));
+        assert!(packets[1].starts_with("a=T,f=100,t=d,"));
+        for packet in packets.iter().skip(2) {
             assert!(packet.starts_with("m="));
             assert!(!packet.contains("a=T"));
             let (_, data) = packet.split_once(';').unwrap();

src/render/protocols/plot/display_list.rs

@@ -133,6 +133,65 @@ pub(super) fn build_body_display_list(
     list
 }
 
+pub(super) fn build_body_base_display_list(
+    scene: &PlotScene,
+    viewport: PlotBounds,
+    dimensions: PlotDimensions,
+    theme: PlotTheme,
+    text: TextMetrics,
+) -> PlotDisplayList {
+    let mut list = PlotDisplayList {
+        dimensions,
+        background: theme.background,
+        text,
+        commands: Vec::new(),
+    };
+
+    if scene.series.is_empty() {
+        return list;
+    }
+
+    let _bounds = viewport.normalized();
+    let layout = body_layout_for(dimensions, text);
+    push_frame(&mut list, layout.area, theme.axis);
+    push_grid_lines(&mut list, layout.area, theme.grid);
+    list
+}
+
+pub(super) fn build_body_marks_display_list(
+    scene: &PlotScene,
+    kind: PlotKind,
+    viewport: PlotBounds,
+    dimensions: PlotDimensions,
+    theme: PlotTheme,
+    text: TextMetrics,
+) -> PlotDisplayList {
+    let mut list = PlotDisplayList {
+        dimensions,
+        background: Rgba([0, 0, 0, 0]),
+        text,
+        commands: Vec::new(),
+    };
+
+    if scene.series.is_empty() {
+        return list;
+    }
+
+    let bounds = viewport.normalized();
+    let layout = body_layout_for(dimensions, text);
+    for (series_index, series) in scene.series.iter().enumerate() {
+        let color = Rgba(theme.strokes[series_index % theme.strokes.len()]);
+        match kind {
+            PlotKind::Line => push_line_series(&mut list, series, bounds, &layout.area, color),
+            PlotKind::Scatter => {
+                push_scatter_series(&mut list, series, bounds, &layout.area, color)
+            }
+        }
+    }
+
+    list
+}
+
 fn push_frame(list: &mut PlotDisplayList, area: PlotArea, color: Rgba<u8>) {
     push_line(
         list,