feat: Implement real-time streaming output with interactive multi-node UI

📋 Overview

Implement real-time command output streaming capability for SSH execution, inspired by PR #37 but with enhanced multi-node UI support. Currently, bssh waits for commands to complete before showing any output, making it difficult to monitor long-running operations across multiple nodes.

Note: This issue has been significantly updated to reflect the current implementation status as of v1.4.0. Most planned features have been implemented.

✅ Implementation Status Summary

Feature	Status	Location
Core Streaming API	✅ Complete	src/ssh/tokio_client/channel_manager.rs
Independent Stream Management	✅ Complete	src/executor/stream_manager.rs
Simple Output Modes (--stream, --output-dir)	✅ Complete	src/cli.rs, src/executor/output_mode.rs
Interactive TUI (ratatui)	✅ Complete	src/ui/tui/
Summary View	✅ Complete	src/ui/tui/views/summary.rs
Detail View	✅ Complete	src/ui/tui/views/detail.rs
Split View	✅ Complete	src/ui/tui/views/split.rs
Diff View	✅ Complete	src/ui/tui/views/diff.rs
Progress Parsing	✅ Complete	src/ui/tui/progress.rs
TTY Detection	✅ Complete	atty dependency
Memory Overflow Protection	✅ Complete	RollingBuffer with 10MB limit
Comprehensive Tests	🚧 Partial	Unit tests exist, more integration tests needed
Documentation	🚧 Partial	Code documented, ARCHITECTURE.md needs update

🎯 Goals (Updated)

1. Real-time Output Streaming: Enable streaming of stdout/stderr as commands execute ✅ Complete
2. Multi-node Observability: Provide dynamic UI to monitor multiple nodes simultaneously ✅ Complete
3. Independent Stream Management: Maintain separate output streams per node ✅ Complete
4. Backward Compatibility: Maintain existing API and behavior ✅ Complete

🏗️ Architecture (Current Implementation)

Core Streaming Infrastructure ✅

// Implemented in src/ssh/tokio_client/channel_manager.rs

/// Command output variants for streaming
#[derive(Debug, Clone)]
pub enum CommandOutput {
    StdOut(CryptoVec),
    StdErr(CryptoVec),
    ExitCode(u32),
}

impl Client {
    // Existing method (backward compatible) - uses streaming internally
    pub async fn execute(&self, cmd: &str) -> Result<CommandExecutedResult>;

    // Streaming method
    pub async fn execute_streaming(
        &self,
        command: &str,
        sender: Sender<CommandOutput>
    ) -> Result<u32, Error>;

    // Sudo password support
    pub async fn execute_with_sudo(
        &self,
        command: &str,
        sender: Sender<CommandOutput>,
        sudo_password: &SudoPassword,
    ) -> Result<u32, Error>;
}

Independent Stream Management ✅

// Implemented in src/executor/stream_manager.rs

/// Independent output stream for a single node
pub struct NodeStream {
    pub node: Node,
    receiver: mpsc::Receiver<CommandOutput>,
    stdout_buffer: RollingBuffer,  // 10MB max with overflow protection
    stderr_buffer: RollingBuffer,
    status: ExecutionStatus,
    exit_code: Option<u32>,
    closed: bool,
}

/// Manager for coordinating multiple node streams
pub struct MultiNodeStreamManager {
    streams: Vec<NodeStream>,
}

/// Execution status for a node's command
pub enum ExecutionStatus {
    Pending,
    Running,
    Completed,
    Failed(String),
}

TUI Architecture ✅

// Implemented in src/ui/tui/

pub mod app;           // TuiApp, ViewMode
pub mod event;         // Keyboard event handling
pub mod progress;      // Progress bar parsing
pub mod terminal_guard; // RAII terminal cleanup
pub mod views;         // summary, detail, split, diff

pub enum ViewMode {
    Summary,              // Show all nodes status
    Detail(usize),        // Focus on single node
    Split(Vec<usize>),    // Show multiple nodes in panes
    Diff(usize, usize),   // Compare two nodes side-by-side
}

📐 Multi-node UI (Implemented)

CLI Output Modes ✅

# TUI Mode (default when TTY detected)
$ bssh -C production "apt-get update"
# → Opens interactive TUI with real-time monitoring

# Stream Mode (--stream flag)
$ bssh -C prod --stream "command"
[node1] Starting process...
[node2] Starting process...
[node1] Progress: 50%

# File Mode (--output-dir flag)
$ bssh -C prod --output-dir ./logs "command"
# Creates: ./logs/node1_TIMESTAMP.stdout, etc.

# Normal Mode (auto when piped)
$ bssh -C prod "uptime" | grep -v idle

TUI Views (All Implemented) ✅

Summary View (Press Esc from other views)

┌──────────────────────────────────────────────────────────┐
│ Cluster: production - apt-get upgrade                    │
│ Total: 8 • ✓ 3 • ✗ 1 • 4 in progress                    │
├──────────────────────────────────────────────────────────┤
│ [1] node1                ✓ Completed (exit: 0)          │
│ [2] node2                ⟳ [=========     ] 75%         │
│ [3] node3                ⟳ Running...                   │
│ [4] node4                ✗ Exit code: 1                 │
│ [5] node5                ⟳ [==           ] 25%          │
├──────────────────────────────────────────────────────────┤
│ [1-9] Detail  [s] Split  [d] Diff  [q] Quit  [?] Help   │
└──────────────────────────────────────────────────────────┘

Detail View (Press 1-9)

Full output view with scrolling and follow mode toggle (f key).

Split View (Press s)

Shows 2-4 nodes simultaneously in split panes.

Diff View (Press d)

Side-by-side comparison of two nodes.

Keyboard Controls ✅

• 1-9: Jump to node detail view
• s: Enter split view
• d: Enter diff view (select two nodes)
• f: Toggle auto-scroll (follow mode)
• ↑/↓: Scroll output
• ←/→: Switch between nodes in detail view
• Esc: Return to summary view
• ?: Show help overlay
• q: Quit

📦 Dependencies (Current)

# Already implemented in Cargo.toml
ratatui = "0.29"      # TUI framework
crossterm = "0.29"    # Terminal control
atty = "0.2.14"       # TTY detection
tokio = { version = "1.47.1", features = ["full"] }
indicatif = "0.18"    # Progress indicators (parallel mode)

🛠️ Implementation Status

Task 1: Core Streaming API ✅

• Implement execute_streaming() API
• Add CommandOutput enum with StdOut/StdErr/ExitCode
• Implement CommandOutputBuffer for internal use
• Ensure execute() maintains backward compatibility (uses streaming internally)
• Add error type for JoinError
• Add execute_with_sudo() for sudo password handling

Task 2: Independent Stream Management ✅

• Implement NodeStream struct with independent buffering
• Create MultiNodeStreamManager for coordinating streams
• Implement non-blocking polling (poll() method)
• Add per-node state management (ExecutionStatus)
• Handle partial failures gracefully
• Add RollingBuffer with 10MB limit for memory protection

Task 3: Simple Output Modes ✅

• Add --stream flag for interleaved output with [node] prefixes
• Implement --output-dir for per-node file output
• Add TTY detection (auto-enable TUI in terminals)
• Update CLI argument parsing in src/cli.rs
• Implement OutputMode enum (Normal/Stream/File/Tui)

Task 4: Interactive TUI ✅

• Add ratatui and crossterm dependencies
• Implement summary view component
• Add detail view with node switching
• Implement split view mode (2-4 nodes)
• Add diff mode for comparing two nodes
• Implement progress parsing heuristics
• Add keyboard navigation
• Add auto-scroll control (follow mode)
• Implement help overlay (? key)
• Add terminal size validation with error message

Task 5: Testing & Documentation 🚧

• Unit tests for stream management
• TUI integration tests with ratatui's test backend
• Update ARCHITECTURE.md with TUI architecture
• Add usage examples in README.md

🎁 Additional Features Implemented (Beyond Original Scope)

The following features were implemented but not originally planned in this issue:

1. Sudo Password Support (execute_with_sudo())

   • Automatic sudo prompt detection
   • Secure password injection with PTY
   • Multiple sudo prompt handling (up to 10 per session)
   • Buffer size limits for security (64KB)

2. Memory Protection

   • RollingBuffer with configurable max size (10MB default)
   • Automatic old data discard to prevent OOM
   • Overflow logging and warnings

3. Terminal Guard

   • RAII-based terminal cleanup
   • Proper handling of crashes/panics
   • Cursor visibility management

4. Progress Parsing Heuristics

   • Detection of XX% patterns
   • Status message extraction from output
   • Integration with summary view

🎯 Remaining Work

1. Documentation

   • Update ARCHITECTURE.md with new TUI module structure
   • Add TUI screenshots to README
   • Document keyboard shortcuts in help output

2. Testing

   • Add TUI snapshot tests using ratatui's test backend
   • Integration tests for streaming execution
   • Performance tests for large output handling

3. Future Enhancements (Lower Priority)

   • Configurable buffer sizes via CLI/config
   • Output search/filtering within TUI
   • Session recording and playback
   • Per-node selective logging

📁 Current Project Structure (Relevant Files)

bssh/
├── src/
│   ├── cli.rs                    # CLI with --stream, --output-dir flags
│   ├── executor/
│   │   ├── mod.rs                # ParallelExecutor exports
│   │   ├── output_mode.rs        # OutputMode enum
│   │   └── stream_manager.rs     # NodeStream, MultiNodeStreamManager
│   ├── ssh/
│   │   ├── client/
│   │   │   ├── mod.rs            # SshClient exports
│   │   │   ├── command.rs        # execute(), execute_streaming()
│   │   │   └── ...
│   │   └── tokio_client/
│   │       ├── mod.rs            # Client, CommandOutput exports
│   │       ├── channel_manager.rs # CommandOutput, execute_streaming()
│   │       └── ...
│   ├── ui/
│   │   ├── mod.rs                # UI exports
│   │   └── tui/
│   │       ├── mod.rs            # run_tui(), TuiExitReason
│   │       ├── app.rs            # TuiApp, ViewMode
│   │       ├── event.rs          # Keyboard handling
│   │       ├── progress.rs       # Progress parsing
│   │       ├── terminal_guard.rs # RAII terminal cleanup
│   │       └── views/
│   │           ├── mod.rs
│   │           ├── summary.rs
│   │           ├── detail.rs
│   │           ├── split.rs
│   │           └── diff.rs
│   └── commands/
│       └── exec.rs               # Execute command with output modes
└── Cargo.toml                    # ratatui, crossterm, atty dependencies

📚 References

• Original PR #37: https://github.com/lablup/bssh/pull/37
• russh documentation: https://docs.rs/russh/
• ratatui documentation: https://docs.rs/ratatui/
• ratatui examples: https://github.com/ratatui-org/ratatui/tree/main/examples

✅ Acceptance Criteria (Status)

• execute_streaming() API works with single node
• Multi-node execution maintains independent streams per node
• Node switching is instant (no re-fetching of output)
• Each node preserves scroll position and state when switching
• --stream mode works in terminals and pipes
• TUI activates automatically in interactive terminals
• All existing tests pass (backward compatibility)
• New tests cover streaming scenarios and view modes (partial)
• Documentation updated (README, ARCHITECTURE.md) (partial)