Performance Optimizations

This document describes the performance optimizations implemented in napari-hdf5-activity for processing large datasets with high resolution and RGB channels.

---

Table of Contents

1. Overview
2. RGB to Grayscale Conversion
3. Dynamic RAM Management
4. Worker Thread Management
5. Recommended Settings
6. Performance Benchmarks

---

Overview

Processing large HDF5 files with high resolution (e.g., 1920×1080) and RGB channels can be computationally intensive. We have implemented several optimizations to significantly improve processing speed while maintaining memory safety.

Key Optimizations

• Vectorized RGB→Grayscale conversion: 10-100× faster than frame-by-frame processing
• Dynamic RAM-based task queueing: Automatically adapts to available system memory
• Improved worker thread management: Prevents duplicate analyses and ensures clean shutdown

---

RGB to Grayscale Conversion

Problem

Old HDF5 files with RGB channels require conversion to grayscale for movement analysis. The original implementation used frame-by-frame conversion with OpenCV:

# OLD (SLOW): Frame-by-frame conversion
grayscale_stack = np.array(
    [cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY) for frame in image_stack]
)

Issues: - One function call per frame (10,000 frames = 10,000 calls) - Python loop overhead - Memory inefficient (creates temporary lists)

Solution

Vectorized NumPy operation using tensordot:

# NEW (FAST): Vectorized conversion
weights = np.array([0.299, 0.587, 0.114], dtype=np.float32)
grayscale_stack = np.tensordot(image_stack, weights, axes=([3], [0]))
grayscale_stack = grayscale_stack.astype(image_stack.dtype)

Benefits: - Single operation for entire stack - Uses optimized BLAS/LAPACK routines - Memory efficient (no intermediate arrays) - Standard ITU-R 601-2 luma transform (same as cv2.cvtColor)

Performance Impact

Resolution	Frames	Old Method	New Method	Speedup
640×480 RGB	1,000	~5s	~0.5s	10×
1920×1080 RGB	1,000	~15s	~1s	15×
1920×1080 RGB	10,000	~150s	~10s	15×

Real-world impact: A 2-hour RGB recording (10,000 frames) that previously took 2.5 minutes to convert now takes only 10 seconds.

---

Dynamic RAM Management

Problem

Processing large files in parallel can exhaust system RAM, leading to: - System slowdown (swapping) - Out-of-memory errors - Unpredictable behavior

The old system submitted all tasks to the processing queue at once, regardless of available RAM.

Solution

Dynamic queue sizing based on available system memory using psutil:

# Calculate available RAM
available_ram_gb = psutil.virtual_memory().available / (1024**3)

# Estimate RAM per chunk
chunk_size_mb = (frame_width × frame_height × channels × dtype_size × chunk_frames) / (1024**2)

# Calculate safe queue size (use 50% of available RAM)
max_queue_size = int((available_ram_gb × 1024 × 0.5) / chunk_size_mb)

Behavior by System

System RAM	Available	Chunk Size	Max Queue	Behavior
64 GB (Workstation)	48 GB	100 MB	~240 tasks	Maximum parallelization
32 GB (High-end)	24 GB	100 MB	~120 tasks	High parallelization
16 GB (Mid-range)	8 GB	100 MB	~40 tasks	Balanced
8 GB (Low-end)	2 GB	100 MB	~10 tasks	Conservative, safe

Benefits: - High-end systems: Utilize full RAM for maximum speed - Low-end systems: Prevent swapping and crashes - Automatic adaptation: No manual configuration needed - Safe fallback: Conservative defaults if psutil unavailable

Queue Management Strategy

Processing Pipeline:
┌─────────────────────────────────────────┐
│ HDF5 File (1000 chunks)                 │
└──────────────┬──────────────────────────┘
               │
               ▼
    ┌──────────────────────┐
    │  Active Queue        │  ← Dynamic size (e.g., 40 tasks)
    │  [Task 1] [Task 2]   │
    │  [Task 3] [Task 4]   │
    │      ...             │
    └──────────────────────┘
               │
               ▼
    ┌──────────────────────┐
    │ Process Pool (4)     │
    │ [P1] [P2] [P3] [P4] │
    └──────────────────────┘
               │
               ▼
         Results merged

How it works: 1. Submit initial batch of tasks (based on available RAM) 2. As tasks complete, submit new tasks one-by-one 3. Maintain constant RAM usage 4. Process continues until all chunks processed

---

Worker Thread Management

Problem

Analysis runs in background threads. Previous implementation had issues: - Multiple analyses could run simultaneously (progress bar confusion) - Stop button didn't cleanly terminate workers - Worker references not properly cleared - Callbacks could fire after stop requested

Solution

1. Prevent Duplicate Analyses

def run_analysis(self):
    # Check if analysis is already running
    if hasattr(self, 'current_worker') and self.current_worker is not None:
        self._log_message("⚠️ Analysis already running!")
        return
    # ... continue with analysis

2. Clean Worker Shutdown

def stop_analysis(self):
    # Set cancellation flag
    self._cancel_requested = True

    # Disconnect signals to prevent callbacks
    if self.current_worker is not None:
        self.current_worker.returned.disconnect()
        self.current_worker.errored.disconnect()
        self.current_worker.finished.disconnect()

        # Clear worker reference
        self.current_worker = None

3. Proper Cleanup

def _analysis_done(self):
    # Always clear worker reference when done
    self.current_worker = None
    self._cancel_requested = False
    # ... reset UI state

Benefits: - Single analysis at a time (clear progress indication) - Clean stop/restart workflow - No zombie threads - No orphaned callbacks

---

Recommended Settings

For External SSD with Limited RAM (8-16 GB)

Parameter	Recommended	Why
Chunk Size	50-100 frames	Balance between I/O and RAM
Processes	2-3	Avoid RAM exhaustion
Expected RAM	~300-600 MB	Manageable for most systems

For High-End System (32+ GB RAM, NVMe SSD)

Parameter	Recommended	Why
Chunk Size	100-200 frames	Maximize sequential I/O
Processes	4-6	Full CPU utilization
Expected RAM	~2-4 GB	Plenty of headroom

For HDD Storage

Parameter	Recommended	Why
Chunk Size	150-200 frames	Minimize seek operations
Processes	2-3	I/O bound, not CPU bound

Note: The dynamic RAM management system automatically adjusts queue size, so these are starting recommendations. The system will adapt to your specific hardware.

---

Performance Benchmarks

Real-World Test Results

Test Configuration: - File: Nematostella timelapse (IR-only, grayscale) - Resolution: 1024×1224 pixels - Frames: 6,103 (~8.5 hours at 5s intervals) - ROIs: 6 - Storage: External SSD - System: Mid-range (limited RAM)

Results by Configuration

Chunk Size	Processes	Time	FPS	Speedup	Efficiency
20	1	108.7s	56.2	1.00×	100%
20	2	57.3s	106.5	1.90×	95%
20	4	34.2s	178.3	3.18×	80%
50	1	101.2s	60.3	1.07×	107%
50	2	55.2s	110.5	1.97×	99%
50	4	32.5s	188.1	3.35×	84% ← Best
100	1	98.9s	61.7	1.10×	110%
100	2	54.1s	112.9	2.01×	101%
100	4	33.9s	180.3	3.21×	80%

Key Findings:

✅ Optimal Configuration (External SSD): - Chunk Size: 50 frames - Processes: 4 - Processing time: 32.5 seconds (vs 108.7s sequential) - Speedup: 3.35× (84% of theoretical 4× maximum) - Processing rate: 188 frames/second

✅ RAM-Safe Configuration: - Chunk Size: 100 frames - Processes: 2 - Processing time: 54.1 seconds - Speedup: 2.01× (still very good!) - Lower memory footprint

✅ Chunk Size Impact: - Larger chunks (50-100) slightly faster than small chunks (20) - Improvement: ~7-10% due to reduced I/O overhead - Optimal: 50 frames for external SSD

Scalability Analysis

Speedup by Process Count: - 1→2 processes: 1.90× speedup (95% efficient) - 1→4 processes: 3.18× speedup (80% efficient)

Excellent scaling! The ~20% overhead at 4 processes is typical and acceptable, caused by: - Process communication overhead - I/O contention - Dynamic queue management

CPU Utilization

Configuration	CPU Usage	Notes
1 process	~25%	Single core utilized
2 processes	~50%	Good parallel efficiency
4 processes	~80%	Excellent utilization

Much improved from old implementation (20-40% usage)!

Recommended Workflow

1. First Run: Use default settings (chunk=50, processes=auto)
2. Monitor: Check RAM usage and CPU load
3. Adjust if needed:
4. High RAM usage (>80%): Reduce chunk size or processes
5. Low CPU usage (<50%): Increase chunk size for better I/O
6. External SSD: chunk=50-100
7. HDD: chunk=150-200

---

Technical Details

RGB to Grayscale Formula

Both methods use the ITU-R 601-2 luma transform:

Y = 0.299*R + 0.587*G + 0.114*B

Why these weights? - Based on human perception (green is brightest, blue is darkest) - Standard used by cv2.cvtColor - Ensures consistent results with other tools

Memory Estimation

# Single frame memory
frame_memory = width × height × channels × bytes_per_pixel

# Chunk memory
chunk_memory = frame_memory × frames_per_chunk

# Total process memory (worst case)
total_memory = chunk_memory × num_processes × 2  # ×2 for processing overhead

Example (1920×1080 RGB, uint8, chunk=100, 4 processes):

Frame: 1920 × 1080 × 3 × 1 = 6.2 MB
Chunk: 6.2 MB × 100 = 620 MB
Total: 620 MB × 4 × 2 ≈ 5 GB (peak usage)

Dynamic Queue Calculation

# Conservative estimate: use 50% of available RAM
usable_ram = available_ram × 0.5

# How many chunks fit?
max_chunks = usable_ram / chunk_memory

# Ensure minimum parallelism
max_queue = max(4, min(max_chunks, total_chunks))

---

Troubleshooting

High RAM Usage

Symptoms: System slowing down, RAM >90%

Solutions: 1. Reduce chunk size: 100 → 50 frames 2. Reduce processes: 4 → 2 3. Close other applications 4. Consider processing in batches

Low CPU Usage (<40%)

Symptoms: Long processing time, disk activity high

Solutions: 1. Increase chunk size: 50 → 100 frames 2. Check storage speed (HDD vs SSD) 3. Reduce number of processes (I/O bound)

Analysis Won't Stop

Symptoms: Stop button clicked but analysis continues

Expected behavior: Analysis stops after current chunk completes (1-30 seconds depending on chunk size)

If it doesn't stop: 1. Wait for current chunk to finish 2. Check log for "STOP requested" 3. Restart application if frozen

"Analysis already running" Message

Cause: Trying to start new analysis while one is running

Solution: Wait for current analysis to finish or click Stop first

---

Future Optimizations

Potential areas for further improvement:

1. GPU acceleration for RGB→Gray conversion (OpenGL/CUDA)
2. Compression-aware chunking for HDF5 files
3. Incremental processing for very large files (>100 GB)
4. Smart chunk sizing based on file structure analysis
5. Parallel I/O with memory-mapped files

---

References

• NumPy tensordot documentation: https://numpy.org/doc/stable/reference/generated/numpy.tensordot.html
• ITU-R BT.601-7: https://www.itu.int/rec/R-REC-BT.601
• psutil documentation: https://psutil.readthedocs.io/
• Python ProcessPoolExecutor: https://docs.python.org/3/library/concurrent.futures.html

---

Last Updated: 2025-12-28 Version: 1.0