Our Databricks-based CDC processing job was lagging during traffic spikes and wasting money during quiet periods. Here’s how we taught it to scale itself.