Files

kevin-asprec 132290957c feat(phase-5): on-prem sync poll hardening + stale job reclaim

Harden the Hub-side polling protocol for on-prem TapTrack agents:
- sync/poll: accept job_failed_ids (retry/fail on-prem delivery failures)
- sync/poll: deduct credits + write ledger when on-prem reports sent jobs
- sync/poll: return feature_flags (tier-based) + suspended flag in config
- sync/poll: skip job dispatch for suspended/expired schools
- sms_jobs: add delivered_via (pull|push) + processing_started_at columns
- tasks/sms: new sms.reclaim_stale_jobs task resets processing→pending if on-prem
  goes offline (jobs stuck >5 min), enabling Celery push fallback
- tasks/sms: tag Celery-sent jobs as delivered_via='push'
- worker: schedule reclaim_stale_jobs every 5 minutes
- migration: 001_phase5 adds delivered_via + processing_started_at to sms_jobs

2026-03-16 12:30:37 +08:00

2.5 KiB

Raw Blame History

Phase 05: On-Prem SMS Polling Agent (Hub Side)

Status: Complete Completed: 2026-03-16

Goal

Harden the Hub's side of the on-prem polling protocol:

Credit deduction when on-prem reports delivered jobs
Failed job reporting (on-prem couldn't send → Hub handles retry)
Feature flags returned with every poll
Suspension flag in config response
Graceful degradation: stale processing jobs reclaimed back to pending if on-prem goes offline

Plan

5-01: sync/poll enhancements + credit deduction (Hub side)

Changes to backend/app/routers/sync.py:

Accept job_failed_ids: list[str] in poll body — increment retry_count, mark failed at 5
Deduct 1 SMS credit per job in report_sent_ids, write SmsCreditLedger entries
Fire send_low_credit_alert if credits fall below threshold after deductions
Add feature_flags dict to config response (tier-based, reusing _tier_features() from licenses.py)
Add suspended: bool to config response
Return 403 with reason field if license is expired (not just revoked)

Changes to backend/app/models/sms.py:

Add delivered_via: Mapped[str | None] column ("pull" | "push" | None)

Changes to backend/app/tasks/sms.py:

Add reclaim_stale_jobs() task — reset processing jobs older than 5 minutes back to pending (handles on-prem going offline mid-cycle)
Mark jobs sent by Celery push path as delivered_via = "push"

Changes to backend/app/worker.py:

Schedule sms.reclaim_stale_jobs every 5 minutes

5-02: Alembic migration for delivered_via column

Add delivered_via VARCHAR(10) nullable to sms_jobs table.

Architecture

On-prem TapTrack (every 30s):
  POST /api/sync/poll
  Body: {
    license_key: "TTUB-XXXXX",
    report_sent_ids: ["uuid1", "uuid2"],   ← jobs on-prem successfully sent
    job_failed_ids:  ["uuid3"]             ← jobs on-prem could NOT send
  }

Hub response:
  {
    sms_jobs: [...],           ← up to 50 pending jobs to send
    config: {
      sms_sender_name: "...",
      sms_credits: 47.0,
      school_status: "active",
      suspended: false,
      feature_flags: { sms: true, reports: true, ... }
    }
  }

Credit flow (pull path):
  On-prem sends → reports sent_ids next poll → Hub deducts 1 credit per job
  (NOT deducted when job is dispatched — only when confirmed sent)

Graceful degradation:
  sms.reclaim_stale_jobs (every 5min):
    UPDATE sms_jobs SET status='pending'
    WHERE status='processing' AND updated_at < now() - 5min
  → If on-prem dies mid-poll, jobs return to Celery push queue

2.5 KiB Raw Blame History