Job Monitoring, Multi-Site Scraping, and Production Polish

Week 36

#Autoscroll Recorder

  • Added "unknown" status detection for jobs that get lost in the pipeline, making it easier to spot and retry failed jobs.
  • Improved job tracking by switching from job IDs to SQS message IDs for more reliable ECS task matching.
  • Added batch job tracking and repository cleanup features.
  • Built a reusable jobs table component for consistent UI across different pages.
  • Launched a full API system for programmatic job creation, enabling automated workflows.
  • Implemented type-safe API key management with proper authentication.
  • Fixed authentication issues with X-API-Key header support for Vercel compatibility.

UX and recording improvements:

  • Fixed expired presigned URLs on job details pages.
  • Enabled retry for jobs with unknown status.
  • Removed unnecessary page reloads after bulk job refresh.
  • Improved configuration settings UI with better documentation.
  • Hidden the cursor in video recordings using FFmpeg's -draw_mouse option.
  • Prevented cookie banners from hiding entire HTML elements.
  • Implemented URL-based filename generation with date partitioning for better organization.

#Inspiration Index

  • Migrated to a new multi-site scraper architecture using the Firecrawl SDK.
  • Launched an Awwwards scraper MVP with daily cron scheduling and backfill capabilities.
  • Separated scraping logic from recording workflows for better modularity.
  • Added type-safe enums and improved submission handling.
  • Implemented proper lock release on script failures with enhanced logging.
  • Implemented shared ECR repository infrastructure across environments.
  • Updated production configuration for deployment readiness.
  • Added ECS task tagging for improved job tracking.
  • Fixed ES module compatibility issues for Vercel deployments.