Skip to main content

MagicLogger Architecture

Overview​

MagicLogger is a high-performance, feature-rich logging library for TypeScript/JavaScript applications. Our goal is to provide styled logs in production environments with minimal performance overhead through efficient I/O patterns and smart caching.

Core Components​

1. Logger Classes​

Logger (Default)​

  • Purpose: Synchronous logging with optional buffering for balanced performance
  • Use Cases: General-purpose logging, development
  • Implementation: Direct transport calls with optional buffering

SyncLogger​

  • Purpose: Explicit blocking I/O for guaranteed delivery
  • Use Cases: Audit logging, debugging, crash-resilient logging
  • Trade-offs: Blocks event loop, guarantees write completion

AsyncLogger​

  • Purpose: Non-blocking logging with immediate dispatch
  • Implementation:
    • Immediate dispatch to transports (no batching at logger level)
    • Optional worker threads (enable with worker.enabled: true)
    • Style processing in main thread by default (~0.01ms overhead)
    • Direct transport dispatch when workers disabled
  • Use Cases: High-throughput applications, web servers
  • Trade-offs:
    • Main thread: Lower latency, simpler architecture
    • With workers: Better for CPU-intensive styling, adds IPC overhead

2. Transport System​

Transport Interface​

interface Transport {
  name: string;
  enabled: boolean;
  log(entry: LogEntry): void | Promise<void>;
  flush?(): void | Promise<void>;
  close(): void | Promise<void>;
  shouldLog(entry: LogEntry): boolean;
  logBatch?(entries: LogEntry[]): void | Promise<void>;
  init?(): void | Promise<void>;
}

Built-in Transports​

  • ConsoleTransport: Outputs to stdout/stderr with colors
  • FileTransport: High-performance async file writes using sonic-boom (default, alias for AsyncFileTransport)
  • AsyncFileTransport: Explicit async file writes using sonic-boom
  • SyncFileTransport: Synchronous file writes with intelligent batching
  • WorkerFileTransport: Worker thread-based file transport for CPU-intensive processing
  • HTTPTransport: Sends logs to HTTP endpoints with batching
  • NullTransport: Discards logs (for benchmarking)

Transport Manager​

  • Coordinates multiple transports
  • Handles batching and routing
  • Implements backpressure handling
  • Provides metrics and health monitoring

3. Dispatch Architecture​

Immediate Dispatch Strategy​

MagicLogger uses an immediate dispatch architecture for optimal performance:

  1. Logger Level (AsyncLogger):

    • Immediate dispatch to transports
    • No batching at logger level
    • Minimal overhead with timestamp caching

  2. Transport Level:

    • Each transport implements its own batching based on I/O needs
    • FileTransport: Uses sonic-boom's internal 4KB buffer
    • HTTPTransport: Batches 100 entries or 5s timeout for network efficiency
    • ConsoleTransport: No batching (immediate output)

This architecture ensures minimal latency while allowing transports to optimize their I/O patterns.

3. Async I/O Architecture​

Default: FileTransport with sonic-boom​

Main Thread                    Async I/O
    │                               │
    ├─ Logger                     ├─ sonic-boom
    │   ├─ Format Entry           │   ├─ Internal Buffer
    │   └─ Call Transport         │   ├─ Auto-flush at minLength
    │                             │   └─ fs.write() (non-blocking)
    └─ FileTransport              │
        ├─ logSync()              └─ File System
        └─ Direct to sonic-boom       └─ Disk

Default behavior: Logger uses FileTransport (AsyncFileTransport) with sonic-boom for async I/O - provides the best performance for production applications

Optional: WorkerFileTransport for CPU-intensive workloads​

Main Thread                    Worker Thread Pool
    │                               │
    ├─ Logger                     ├─ Worker 1
    │   ├─ Format Entry           │   ├─ Process batch
    │   └─ Send to Worker         │   └─ Write to file
    │                             │
    └─ WorkerTransport            ├─ Worker 2 (if poolSize > 1)
        ├─ Batch logs                 ├─ Process batch
        └─ IPC transfer               └─ Write to file

Worker threads are optional: Only use WorkerTransport when you need true parallelism for CPU-intensive processing

Worker Communication Protocol (When using WorkerTransport)​

  • INIT: Initialize worker with transport config
  • LOG_BATCH: Send batch of logs to process
  • FLUSH: Force flush buffered logs
  • SHUTDOWN: Graceful worker termination
  • ACK: Acknowledge batch processing
  • METRICS: Performance metrics updates

Note: This protocol only applies when explicitly using WorkerTransport, not the default AsyncFileTransport

4. MAGIC Schema​

The MAGIC (Metadata And Graphics In Console) schema enables portable styled logs:

interface LogEntry {
  id: string;                    // Unique identifier
  timestamp: string;              // ISO 8601 timestamp
  timestampMs: number;            // Unix timestamp in ms
  level: LogLevel;                // Log severity
  message: string;                // Plain text message
  styles?: Array<[number, number, string]>; // Style ranges [start, end, style]
  context?: Record<string, any>;  // Structured metadata
  tags?: string[];                // Categorization tags
  loggerId?: string;              // Logger instance ID
  error?: {                       // Error information
    name: string;
    message: string;
    stack?: string;
    code?: string | number;
  };
}

Performance Characteristics​

Synchronous Logging​

  • Throughput: ~147K ops/sec plain text, ~50K ops/sec with styles
  • Latency: 0.006ms average blocking time
  • Memory: Minimal buffering
  • Reliability: Guaranteed delivery (blocks until written)

Asynchronous Logging​

  • Throughput: ~164K ops/sec plain text, ~120K ops/sec with styles
  • Latency: 0.006ms average (non-blocking, 0.004ms P50)
  • Memory: Minimal with sonic-boom buffering
  • Reliability: Best-effort, requires graceful shutdown for guarantee
  • Note: Styled async significantly outperforms sync styled (120K vs 50K ops/sec)

Architecture Benefits​

Default Async (sonic-boom)​

  • Non-blocking: Main thread never blocks on I/O
  • Efficient buffering: Automatic flush at configurable thresholds
  • Low overhead: No IPC or thread management costs
  • Production-ready: Battle-tested in Pino ecosystem

Optional Worker Threads (WorkerTransport)​

  • True parallelism: CPU-intensive operations run in parallel
  • Isolation: Transport failures don't affect main thread
  • Scalability: Pool size adjustable based on workload
  • Use cases: Heavy transformations, encryption, compression

Recommendation: Start with default AsyncFileTransport. Only use WorkerTransport if you have specific CPU-intensive requirements that benefit from parallelism.

Worker Thread Considerations​

Based on Node.js best practices:

When to use worker threads:

  • CPU-intensive transformations (encryption, compression)
  • Complex log formatting requiring heavy computation
  • Isolation requirements (untrusted log processing)

When NOT to use worker threads:

  • Simple I/O operations (file writes, network requests)
  • Basic log formatting and styling
  • Low-volume logging scenarios

Performance trade-offs:

  • Worker creation overhead: ~10-50ms per worker
  • IPC overhead: ~0.1-0.5ms per message batch
  • Memory overhead: ~10MB per worker thread
  • Optimal worker count: Number of CPU cores (typically 2-4)

For most logging scenarios, the default async I/O without workers provides the best balance of performance and simplicity.

Real-World Performance Comparison​

LoggerArchitectureThroughput (ops/sec)Avg LatencyUse Case
Pino (Plain)Async I/O226,0460.004msHigh-throughput, minimal overhead
MagicLogger (Async+Styled)Async I/O + Cache163,3500.006msStyled production logging
Winston (Styled)Multi-stream153,4480.006msFeature-rich ecosystem
MagicLogger (Async)Async I/O127,4020.007msNon-blocking production
MagicLogger (Sync)Direct I/O67,8030.014msGuaranteed delivery
MagicLogger (Sync+Styled)Direct I/O + Styles24,8560.040msInteractive CLI tools

Key insights:

  • Async styled (163K) outperforms sync plain (68K) by 2.4x
  • Styling overhead: 63% in sync mode, -28% in async (faster due to better batching)
  • All metrics from real file I/O with production-like payloads

Design Decisions​

1. Async I/O Strategy​

  • Default: sonic-boom for efficient async file I/O
  • Rationale: Proven performance, minimal overhead, no IPC costs
  • Optional: WorkerTransport for CPU-intensive workloads requiring parallelism

2. Batching Strategy​

  • Default: 1000 entries or 10ms timeout
  • Rationale: Balance between syscall reduction and latency
  • Tunable: Via bufferSize and flushInterval options
  • sonic-boom: Internal buffering with automatic flush at minLength

3. Serialization Format​

  • Choice: JSON with MAGIC extensions
  • Rationale: Universal compatibility, structured data
  • Trade-off: Larger payload vs. binary formats

4. Style Processing Architecture​

Where Styling Happens​

Default (Workers OFF - Recommended):

  • Style extraction occurs in the MAIN THREAD
  • Uses TextStyler.parseBracketsWithExtraction() to parse <style>text</> markup
  • Produces { plainText, styles: [[start, end, style], ...] }
  • Result stored in LogEntry for MAGIC schema compliance
  • Performance: ~0.012ms overhead per styled log (acceptable)

With Workers Enabled (Optional):

  • When worker.enabled: true, style extraction moves to WORKER THREAD
  • AsyncLoggerWorker processes styles after receiving batch
  • Beneficial for heavy styling workloads (4x faster for complex styles)
  • Adds IPC overhead for simple logs (~137% slower)

Performance Characteristics (Measured)​

  • Main thread styling: ~0.012ms per styled log (679% overhead vs plain)
  • Simple logs: 888K ops/sec without workers, 375K with workers
  • Styled logs: 95K ops/sec without workers, 405K with workers (4x faster)
  • Optimization: Fast-path detection for non-styled text
  • Recommendation: Use workers only for heavy styling workloads

Memory Management​

Buffer Limits​

  • Batch Buffer: Max 10,000 entries (configurable)
  • Worker Queue: Max 10 concurrent operations per worker
  • Transport Buffer: Transport-specific limits

Backpressure Handling​

  1. Monitor worker utilization
  2. Drop logs when over capacity
  3. Emit warning events
  4. Metrics tracking for monitoring

Error Handling​

Worker Failures​

  • Automatic fallback to setImmediate mode
  • Error events emitted to main thread
  • Graceful degradation

Transport Failures​

  • Individual transport isolation
  • Retry logic (transport-specific)
  • Error aggregation and reporting

Security Considerations​

Log Sanitization​

  • Redaction of sensitive fields
  • PII detection and masking
  • Configurable sanitization rules

Resource Limits​

  • Maximum message size
  • Rate limiting support
  • Memory usage caps

Future Enhancements​

Planned Features​

  1. Compression: zstd/gzip for large batches
  2. Streaming: Server-sent events for real-time logs
  3. Clustering: Multi-process coordination
  4. Tracing: OpenTelemetry integration

Performance Optimizations​

Lock-Free Ring Buffer (Experimental)​

  • SharedArrayBuffer: Zero-copy data transfer between threads
  • Atomic Operations: Lock-free synchronization using Atomics API
  • 64KB Ring Buffer: Pre-allocated circular buffer, no GC pressure
  • 150K+ ops/sec: Target performance with async logging
  • Enable with: worker: { enabled: true, useRingBuffer: true }
  • Note: Requires worker-thread.js to be deployed (coming soon)

Timestamp Precision​

  • SyncLogger: Uses Date.now() for each log (millisecond precision)
  • AsyncLogger: Uses performance.now() for microsecond precision
    • Ensures unique timestamps even in tight loops
    • Prevents timestamp collision in high-throughput scenarios
    • Base timestamp updated every second to prevent drift

Other Optimizations​

  1. Memory Pools: Pre-allocated batch arrays reduce GC pressure
  2. Micro-batching: setTimeout(0) batching for optimal throughput
  3. Zero-Copy Buffers: Direct I/O operations where possible
  4. Smart Caching: Property access caching in hot paths

Best Practices​

For Maximum Throughput (150K+ ops/sec)​

const logger = new AsyncLogger({
  worker: {
    enabled: true,         // Enable workers
    useRingBuffer: true,   // Use lock-free ring buffer
    poolSize: 1,           // Ring buffer uses single worker
    batchSize: 100,        // Optimized batch size
    flushInterval: 10      // Quick flushes
  },
  enableMetrics: true      // Monitor performance
});
// Achieves 150K+ ops/sec with zero-copy ring buffer

For Low Latency (0.003ms avg)​

const logger = new AsyncLogger({
  worker: {
    enabled: false,        // No workers = lowest latency (default)
  },
  buffer: {
    size: 10,              // Small batches
    flushInterval: 1       // Immediate flushes
  }
});
// Achieves 0.003ms average latency with 301K ops/sec

For Reliability​

const logger = new SyncLogger({
  file: './audit.log',     // Persistent storage
  forceFlush: true         // fsync after each write
});

Monitoring​

Metrics Collection​

logger.on('metrics', (metrics) => {
  console.log('Logs processed:', metrics.totalLogs);
  console.log('Worker utilization:', metrics.workerUtilization);
  console.log('Dropped logs:', metrics.droppedLogs);
});

Health Checks​

const health = {
  workers: logger.workers.length,
  pending: logger.batch.length,
  metrics: logger.getMetrics()
};

Testing​

Unit Tests​

  • Mock transports for isolation
  • Deterministic worker behavior
  • Error injection

Integration Tests​

  • Real worker threads
  • File I/O verification
  • Performance benchmarks

Load Tests​

# Generate high load
npm run test:load -- --rate=10000 --duration=60s

# Monitor metrics
npm run test:metrics -- --watch

Performance Characteristics​

Logger TypeThroughputLatencyMemoryBest For
Logger~120K/s<1msLowGeneral use
SyncLogger~53K/sImmediateLowestGuaranteed delivery
AsyncLogger (no workers)~148K/s<7msLowHigh throughput
AsyncLogger (workers)~100K/s<10msModerateCPU-intensive styling

Performance Optimizations​

Timestamp Caching​

  • Optimization: Cache Date.now() for 10 consecutive logs
  • Impact: Reduces syscall overhead

Fast Path Optimization​

  • Optimization: Check for '<' character before style parsing
  • Impact: ~0.01ms saved per plain text log

Object Pre-sizing​

  • Optimization: Pre-allocate all properties with undefined
  • Impact: Faster object creation with V8 optimization

Immediate Dispatch​

  • Optimization: Direct dispatch to transports without batching
  • Impact: Minimal latency for log delivery

Conclusion​

MagicLogger's architecture balances performance, reliability, and developer experience. The immediate dispatch architecture ensures minimal latency, while the MAGIC schema enables rich, portable logging across platforms. Transport-level batching provides optimal I/O performance without sacrificing responsiveness.