Analytics Overhaul, Smarter Crawling, and a Big Migration Week

This week was defined by big architectural moves and reliability wins. The CDP got a full framework migration and a completely redesigned analytics reporting system, while the recorder pipeline saw major stability and UX improvements across both the API and web interface.

Week 78 repos · 53 commits

Silo CDP

The biggest change this week was a full migration to React Router 7 with comprehensive type-safety improvements — a significant architectural upgrade that modernizes the foundation of the platform.

On top of that, the analytics reporting system got a major overhaul:

  • New password-protected analytics reports with an outcomes-first template
  • v2 and v3 report generation with enhanced SQL, comparison support, and improved data collection
  • Several new analytics report configurations added
  • Dark mode got a proper overhaul — custom typography, a cleaner theme system, and consistent component styling
  • CDN paths centralized and the dashboard restructured for better maintainability
  • Server-side bundling issues resolved, including package compatibility fixes
  • Node.js upgraded to v24 LTS
  • Environment variables split into service-specific modules for cleaner configuration

Autoscroll Recorder

API

  • Broken page detection added — recordings now abort immediately when a broken page is detected rather than completing a bad job
  • Mobile experience improved — scroll distance is now device-aware
  • GPU task failover enabled across multiple availability zones for better resilience
  • Service token race conditions fixed
  • Lambda runtime upgraded to Node.js 24
  • Cost estimation tooling added and bot detection improved

Web

  • Batch job cancellation added for active jobs — major workflow improvement for admins
  • CloudWatch logs access added for admin users, with the button now visible even without a task ARN
  • Job status polling now updates all job fields, not just distributions
  • Admin debugging tools relocated to the job details section for a cleaner UX

Link Scraper

  • Four new link sources added (including a design-focused source), with advisory-locked parallel discovery and dry-run testing
  • Round-robin source selection replaced time-based sorting with Fisher-Yates shuffling for more uniform distribution
  • Domain-based rate limiting implemented with run locks and temporary-down propagation
  • Tristate URL liveness tracking added with improved dead/temporary detection
  • Per-device recorder submission system introduced with a dedicated tracking table
  • Recorder now skips fully-submitted websites to avoid redundant work

Inspiration Index Pipeline

  • Gemini 2.5 Flash Lite adopted for frame analysis — faster and more cost-efficient
  • Transient 503 errors from the Google Files API now handled gracefully with retries
  • Video quality validation added before MediaConvert processing
  • Timeout protection and retry logic added to the video frame analysis step
  • Debug URLs (CloudWatch, S3 console) surfaced in error handling and Step Functions state for easier debugging

Inspiration Index App

  • Technology filtering added to the Discover view, making it easier to browse by tech stack

Silo Event Store API

  • IP enrichment caching added, dramatically improving throughput — batch sizes increased to leverage the cache
  • Batch size tuned for cold cache scenarios to prevent overload on startup
  • Database push safety script made interactive
  • Team enrichment made idempotent

This shiplog is partially AI-generated from commit history.