Slashing AWS Infrastructure Expenses by 70% with ECS Spot Draining & EFS Bursting

Running high-availability containerized microservices and stateful persistence layers on AWS can quickly trigger cost anomalies if default managed options like ECS Fargate, EFS Elastic Throughput, and CloudWatch log indexing are used without architectural constraints.

At Resarv, our core platforms process thousands of court reservations, native mobile health telemetry feeds, and millions of enterprise reconciliation transactions daily. To maintain sub-50ms latency while keeping infrastructure overhead low, we engineered a resilient, cost-optimized AWS architecture centered around ECS on EC2 Spot Instances, EFS Bursting Throughput, and OpenObserve OpenTelemetry (OTel) logging.

---

🏗️ 1. Compute & Cluster Architecture: 100% EC2 Spot

Rather than running on expensive AWS Fargate or On-Demand EC2 instances for non-stateful application microservices, our Apps Cluster and Office Apps Cluster operate on 100% Spot Instances using Auto Scaling Group (ASG) Capacity Providers.

# ecs_apps.tf - Capacity Provider Configuration
resource "aws_ecs_capacity_provider" "spot_provider" {
  name = "apps-ec2-spot-provider"

  auto_scaling_group_provider {
    auto_scaling_group_arn = aws_autoscaling_group.ecs_apps_spot.arn
    managed_draining        = "ENABLED"
    managed_termination_protection = "DISABLED"

    managed_scaling {
      maximum_scaling_step_size = 5
      minimum_scaling_step_size = 1
      status                    = "ENABLED"
      target_capacity           = 95
    }
  }
}

Key Spot Optimization Mandates:

1. ENI Trunking Support: All selected EC2 instance types (c6a.xlarge, c7i.xlarge, c7i-flex.xlarge, m6a.xlarge) support Elastic Network Interface (ENI) Trunking, allowing high container density per host.
2. Graceful Spot Draining: Enforced ECS_ENABLE_SPOT_INSTANCE_DRAINING=true in container agent configs. When AWS issues a 2-minute Spot Interruption Notice, ECS automatically sets host status to DRAINING, deregisters tasks from the ALB, and launches replacement tasks on available Spot capacity before node termination.
3. Stateful Isolation: Stateful workloads (production PostgreSQL databases and Valkey cache) are pinned to dedicated 100% On-Demand instances locked to the c7i-flex family to leverage AWS Savings Commitments.

---

💾 2. Resolving EFS Storage Cost Anomalies

During high-I/O persistence operations (PostgreSQL WAL writing, OpenObserve index compaction), using EFS Elastic Throughput mode charges per-GB read/write fees that rapidly trigger AWS cost anomaly alerts.

┌───────────────────────────────────────┬────────────────────────────────────────────────────────┐
│ EFS Throughput Mode                   │ Cost Impact & Behavior                                 │
├───────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ ❌ EFS Elastic Throughput             │ Charges $0.03/GB read and $0.06/GB write.              │
│                                       │ High IOPS trigger massive unexpected monthly spikes.   │
├───────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ ✅ EFS Bursting Throughput            │ Included baseline credit burst mode (0 extra fees).    │
│                                       │ Paired with transition_to_ia = "AFTER_7_DAYS".          │
└───────────────────────────────────────┴────────────────────────────────────────────────────────┘

The Fix:

• Switched all EFS filesystems from elastic to bursting throughput mode.
• Configured Lifecycle Policy to transition_to_ia = "AFTER_7_DAYS".
• Removed transition_to_primary_storage_class = "AFTER_1_ACCESS" to prevent storage price thrashing where accessing archival files forcibly moves them back to primary storage.

---

📊 3. Zero-CloudWatch Logging: OpenObserve & OTel Collector Daemons

Direct CloudWatch log ingestion and log insight queries incur substantial per-GB indexing and retrieval fees. To eliminate these expenses, we deployed OpenTelemetry (OTel) Collector Daemon Services across all ECS hosts.

• Host Network Mode Daemon: OTel collectors run as daemon tasks on all ECS nodes in host network mode, ingesting internal logs, metrics, and traces on local ports 4317 and 4318.
• Single Pane of Glass: Ingested telemetry is batched and forwarded internally to OpenObserve (obs.live.resarv.com).
• 14-Day Expiration Policy: Historical Parquet data is stored on S3 (aws-resarv-env-openobserve-data) with an aggressive 14-day expiration policy (ZO_COMPACT_DATA_RETENTION_DAYS = 14), keeping cloud storage costs at an absolute minimum.

---

📈 Engineering Results

By standardizing on x86_64 Spot capacity providers, EFS bursting throughput, and OTel/OpenObserve logging, our engineering team reduced overall cloud infrastructure spend by over 70% while serving thousands of daily active users with sub-50ms API response times.