Real-Time Event-Driven Content Sync & Fail-Safe Pipeline

Background & Architecture Overview

Previously, the Beautyhaul public article website fetched article listings and search results by querying MySQL directly. Under peak traffic, heavy SQL reads locked database connection pools and degraded response speeds.

To solve this, we migrated the public article data source to Elasticsearch 8, while engineering a dual-layer synchronization architecture:

1. Primary Real-Time Pipeline: AWS SQS + AWS Lambda for instant event-driven article updates.
2. Fail-Safe Backup Pipeline: 2x Daily Golang CLI cron runs to catch any missed updates if SQS or Lambda encounter cloud network glitches.

Dual-Layer Architecture & Data Flow

[ Admin CMS Create/Update ]
           │
  (Instant Event Dispatch)
           ▼
   [ AWS SQS Queue ]
           │
    (Event Trigger)
           ▼
  [ AWS Lambda Worker ] ──(Real-Time Upsert)──► [ Elasticsearch Blog Index ] ◄── [ Public Article Website ]
           │                                            ▲
    (Cloud Glitch/DLQ)                                  │
           ▼                                            │
  [ Fail-Safe CLI Backup ] ──(2x Daily Cron Re-sync)────┘

Technical Solution

1. Elasticsearch as Primary Reader

• Shifted all public article web and mobile API queries from MySQL to Elasticsearch 8.
• Public requests fetch indexed articles in sub-15ms without touching relational databases.

2. Primary Real-Time Sync (AWS SQS + Lambda)

• On every article create or update in the admin CMS, an asynchronous message is published immediately to AWS SQS.
• AWS Lambda consumes the SQS event in real-time, executing a bulk upsert to the Elasticsearch blog index and invalidating matching Redis cache keys.

3. Fail-Safe CLI Backup Pipeline (2x Daily Cron)

• If AWS SQS messages fail or Lambda hits temporary cloud timeout limits, the Golang CLI cron runner acts as a resilient fail-safe backup.
• Morning Run: Full article re-sync batching 100 articles per loop.
• Afternoon Run: Delta article re-sync targeting articles updated during the day.

Measured System Outcomes

• Real-Time Responsiveness: CMS edits propagate to Elasticsearch instantly via AWS SQS + Lambda.
• Data Consistency: 2x daily CLI cron jobs guarantee 100% synchronization even during cloud infrastructure blips.
• Zero DB Locks: Public article readers query Elasticsearch directly, completely isolating MySQL RDS from read traffic.