Datadog Analysis - Clarifications & Corrections

Date: 2025-11-16 Purpose: Address questions and correct misunderstandings from initial analysis

1. Configuration Review: What's Already Supported?

Existing Configuration Parameters (config.go lines 144-191)

Already Configured:

✅ TargetFileSizeMB (line 156) - Default 64 MB, can be changed
✅ CompressionFactor (line 159) - Physical size reduction (0.7 = Snappy default)
✅ DeduplicationFactor (line 158) - Logical size reduction (0.9 default)
✅ MaxStalledWriteMemoryMB (line 178) - OOM threshold (4096 MB default)

NOT Configured:

❌ Write stall delay duration - Currently hardcoded
❌ Block size - Not parameterized at all
❌ Compression/decompression throughput - Not modeled
❌ CPU time for compression - Not in critical path

What I Meant by "Stall Delays"

Current Implementation (simulator.go:657):

stallTime = s.virtualTime + 0.001 // 1ms = 0.001 seconds

The write stall retry interval is hardcoded to 1ms. When writes are stalled (too many immutable memtables), they retry every 1ms until the stall clears.

What I Incorrectly Suggested: I said "write stall delay is 100ms" - this is WRONG. Looking at the code:

The retry interval is 1ms (0.001 seconds)
There is no "100ms delay" anywhere in the stall logic

What SHOULD be configurable:

type SimConfig struct {
    // ...
    WriteStallRetryIntervalMs float64 `json:"writeStallRetryIntervalMs"` // Default: 1.0 ms
}

This would allow testing different stall backpressure behaviors without requiring code changes.

2. Compression Analysis - MAJOR CORRECTION

My Original Claim: "Compression <5% of I/O"

This was MISLEADING and WRONG. Let me explain what I actually calculated vs what it means:

What I Measured (dist-aggr metrics):

Blocks compressed: 18.8 blocks/sec
Blocks decompressed: 83.3 blocks/sec
Assuming 4KB blocks:
- Compression data volume: 75 KB/sec (18.8 × 4KB)
- Decompression data volume: 333 KB/sec (83.3 × 4KB)
Compaction I/O: 10,700 KB/sec

My Flawed Calculation:

75 KB/sec / 10,700 KB/sec = 0.7% (compression)
333 KB/sec / 10,700 KB/sec = 3.1% (decompression)
Conclusion: "<5% of I/O"

Why This Is WRONG:

I measured block operations, not bytes compressed/decompressed
Block operations ≠ I/O volume - These metrics count distinct block cache operations, not total data processed
Compression happens on WRITE PATH, affecting ALL data written (not just 75 KB/sec)
Decompression happens on READ PATH, affecting ALL data read (not just 333 KB/sec)

Correct Analysis from Mork-Storage Profile

Your Profile Data:

Total CPU time: 4m 40s per minute = 280 seconds/minute = 4.67 cores
LZ compression time: ~35 seconds/minute = 0.58 cores
Compression CPU: 35s / 280s = 12.5% of total CPU

This is COMPLETELY DIFFERENT from my "5%" claim.

What This Actually Means

Compression is on the critical path for:

Write Path (Compaction):

Read SST file → Decompress block → Merge keys → Compress block → Write SST file
                 ↑                                  ↑
                 CPU cost                           CPU cost

Duration = max(I/O time, CPU time)
If CPU time ≈ I/O time, then compression DOES matter
Mork shows 12.5% CPU on compression → likely adds latency

Read Path (Point Lookups):
```
Read SST block from disk → Decompress block → Extract key
                           ↑
                           CPU cost (not tiny!)
```
- Block size matters (4KB-64KB typical, configurable)
- Decompressing 16KB-64KB blocks is NOT negligible
- Adds to read latency, especially on cold reads

Implications for Simulator

Current Model (simulator.go:741, 847):

// Compaction duration
duration := (totalInputSize + outputSize) / s.config.IOThroughputMBps

// Flush duration
duration := sizeMB / s.config.IOThroughputMBps

Problem: This assumes I/O is the ONLY cost. But if compression takes 12.5% of CPU time, it's on the critical path.

Better Model:

// Compaction duration
ioTime := (totalInputSize + outputSize) / s.config.IOThroughputMBps
compressionTime := outputSize / s.config.CompressionThroughputMBps  // e.g., 2000 MB/s for Snappy
decompressionTime := totalInputSize / s.config.DecompressionThroughputMBps  // e.g., 3000 MB/s for Snappy
cpuTime := compressionTime + decompressionTime
duration := max(ioTime, cpuTime)  // Bottleneck wins

Why This Matters:

If ioTime > cpuTime: I/O bound (current behavior correct)
If cpuTime > ioTime: CPU bound (current model underestimates duration!)
For Mork (12.5% CPU), likely cpuTime ≈ 0.125 × ioTime → I/O still dominates, but compression adds ~12% overhead

3. Read Path Analysis - Data Source Clarification

My Claim: "Read path is LOW PRIORITY (99.999% writes)"

Which service was this based on?

dist-aggr:

Bytes read: 11,294 bytes/sec (avg)
Bytes written: 949,350,000 bytes/sec (avg)
Read ratio: 11KB / 949MB = 0.0012%
Write ratio: 99.9988%

This is where I got "99.999% writes" - from dist-aggr ONLY.

Mork:

Query rate: 1.2M-4.8M queries/sec
No direct "bytes read" metric available
LRU cache: ~2 GB (full capacity)
This is clearly a READ-HEAVY service!

Correction: Service-Specific Workloads

Service	Workload Pattern	Simulator Priority
dist-aggr	99.999% writes	Write path only (current model OK)
Mork	Read-heavy (queries)	Read path modeling IS important

Takeaway: My conclusion was service-specific, not general. For Mork, read path modeling is HIGH PRIORITY.

4. Mork Profiling Data Analysis

Your Data

Total CPU: 4m 40s/min = 280 seconds = 4.67 cores
LZ compression: 35 seconds/min = 0.58 cores
Compression %: 35s / 280s = 12.5% of CPU

What This Tells Us

Compression is significant (not negligible)
12.5% of CPU time suggests compression is on critical path
RocksDB uses LZ4 (likely) - Fast compression (~2 GB/s single-threaded)

Calculating Compression Impact

Mork Write Throughput (estimated from LSM size):

LSM size: 3.6-5.7 GB (fluctuates)
Avg throughput: ~50-100 MB/s (estimated from size changes)

If compression takes 12.5% of CPU:

4.67 cores × 0.125 = 0.58 cores on compression
At 2 GB/s per core (LZ4): 0.58 cores × 2000 MB/s = 1160 MB/s compression capacity
Write throughput: 50-100 MB/s
Compression capacity >> write throughput → CPU not saturated

But: Compression adds latency to individual operations:

Compressing 64 MB file at 2 GB/s = 32 ms
Reading/writing 64 MB at 500 MB/s = 128 ms
Total: 160 ms (25% increase due to compression)

Conclusion: Compression adds ~20-30% overhead to operation latency, even if I/O is the ultimate bottleneck.

5. Block Size Configuration

Current State: NOT CONFIGURABLE

Searched for "block_size" in simulator/*.go → No results

RocksDB Default: 4KB-16KB (configurable via block_size option)

Why This Matters:

Read Path:
- Point lookup reads 1 key → decompresses entire block
- Larger blocks = more decompression overhead per read
- Smaller blocks = more I/O operations (overhead)
Cache Efficiency:
- Block cache stores compressed or uncompressed blocks
- Block size affects cache hit rate and memory efficiency
Compression Ratio:
- Larger blocks → better compression ratio
- Smaller blocks → worse compression, but lower read amplification

What Should Be Added

type SimConfig struct {
    // ...
    BlockSizeKB              int     `json:"blockSizeKB"`              // SST block size in KB (default 16, typical range 4-64)
    BlockCacheCompressed     bool    `json:"blockCacheCompressed"`     // Store compressed or uncompressed blocks in cache
    BlockCacheSizeMB         int     `json:"blockCacheSizeMB"`         // Block cache size (default 8MB per RocksDB)
}

Impact on Read Latency Model

Current (doesn't exist):

// No read path modeling at all

Proposed:

func (s *Simulator) processReadEvent(event *ReadEvent) {
    // 1. Check block cache
    cacheHit := rand.Float64() < s.config.CacheHitRate

    if cacheHit {
        // Cache hit: only decompression cost (if cache stores compressed blocks)
        if s.config.BlockCacheCompressed {
            latency := float64(s.config.BlockSizeKB) / s.config.DecompressionThroughputMBps
        } else {
            latency := 0.001 // Sub-millisecond (cache hit)
        }
    } else {
        // Cache miss: I/O + decompression
        ioLatency := s.config.IOLatencyMs + (float64(s.config.BlockSizeKB) / s.config.IOThroughputMBps)
        decompressLatency := float64(s.config.BlockSizeKB) / s.config.DecompressionThroughputMBps
        latency := ioLatency + decompressLatency
    }

    // Schedule read completion
    s.queue.Push(NewReadCompleteEvent(s.virtualTime + latency))
}

6. Updated Recommendations

6.1 Add Compression/Decompression Modeling

Priority: HIGH (not "defer" as I originally said)

Rationale:

Mork profile shows 12.5% CPU on compression (not negligible)
Adds 20-30% latency overhead to operations
Critical for accurate compaction duration estimation

Implementation:

type SimConfig struct {
    // ...
    CompressionThroughputMBps   float64 `json:"compressionThroughputMBps"`   // Default: 2000 MB/s (LZ4/Snappy)
    DecompressionThroughputMBps float64 `json:"decompressionThroughputMBps"` // Default: 3000 MB/s (LZ4/Snappy)
}

// Update compaction duration calculation
func calculateCompactionDuration(inputMB, outputMB, ioThroughput, compressThroughput, decompressThroughput float64) float64 {
    ioTime := (inputMB + outputMB) / ioThroughput
    cpuTime := (outputMB / compressThroughput) + (inputMB / decompressThroughput)
    return max(ioTime, cpuTime)  // Bottleneck wins
}

6.2 Add Block Size Configuration

Priority: MEDIUM (needed for read path accuracy)

Add parameters:

BlockSizeKB (default 16 KB)
BlockCacheSizeMB (default 8 MB per RocksDB)
BlockCacheCompressed (default false)

6.3 Implement Read Path Simulation

Priority: HIGH for Mork, LOW for dist-aggr

Requires:

ReadEvent type (doesn't exist yet)
Block cache simulation with hit/miss tracking
Decompression cost calculation
Bloom filter probe simulation (optional)

6.4 Write Stall Configuration

Priority: LOW (current hardcoded 1ms is reasonable)

If needed:

type SimConfig struct {
    // ...
    WriteStallRetryIntervalMs float64 `json:"writeStallRetryIntervalMs"` // Default: 1.0 ms
}

7. Corrected Metrics Interpretation

What I Got Wrong

Claim	Reality	Correction
"Compression <5% of I/O"	Measured block operations, not bytes	Mork profile shows 12.5% CPU on compression
"Write stall delay 100ms"	No such delay exists	Retry interval is 1ms (hardcoded)
"Read path LOW PRIORITY"	Based only on dist-aggr	For Mork (read-heavy), HIGH PRIORITY
"CPU modeling not needed"	Ignored compression overhead	Should add compression/decompression throughput

What I Got Right

Claim	Validation	Status
"I/O is bottleneck"	✅ Mork: compression is 12.5% of CPU, I/O likely 50%+	Correct, but compression matters too
"Production uses 150MB files"	✅ Mork: 4.5GB / 30 files = 150MB avg	Correct
"Write stalls are rare"	✅ dist-aggr: <1µs average stall	Correct
"Cache metrics missing"	✅ block.cache.hit/miss returned zeros	Correct

8. Revised Analysis Summary

CPU vs I/O Bottleneck

Original Conclusion: "I/O is bottleneck, CPU doesn't matter"

Revised Conclusion:

I/O is PRIMARY bottleneck (still true)
Compression adds 20-30% overhead (not negligible!)
Model should include: duration = max(ioTime, cpuTime)

Read Path Importance

Original Conclusion: "Read path LOW PRIORITY (99.999% writes)"