feat(phase-5): on-prem sync poll hardening + stale job reclaim
Harden the Hub-side polling protocol for on-prem TapTrack agents: - sync/poll: accept job_failed_ids (retry/fail on-prem delivery failures) - sync/poll: deduct credits + write ledger when on-prem reports sent jobs - sync/poll: return feature_flags (tier-based) + suspended flag in config - sync/poll: skip job dispatch for suspended/expired schools - sms_jobs: add delivered_via (pull|push) + processing_started_at columns - tasks/sms: new sms.reclaim_stale_jobs task resets processing→pending if on-prem goes offline (jobs stuck >5 min), enabling Celery push fallback - tasks/sms: tag Celery-sent jobs as delivered_via='push' - worker: schedule reclaim_stale_jobs every 5 minutes - migration: 001_phase5 adds delivered_via + processing_started_at to sms_jobs
This commit is contained in:
@@ -1,9 +1,74 @@
|
||||
# Phase 05: On-Prem SMS Polling Agent
|
||||
# Phase 05: On-Prem SMS Polling Agent (Hub Side)
|
||||
|
||||
**Status:** Not started
|
||||
**Status:** Complete
|
||||
**Completed:** 2026-03-16
|
||||
|
||||
## Goal
|
||||
TapTrack polls Hub every 30s for pending SMS jobs; sends them; reports back completion.
|
||||
|
||||
## Plans
|
||||
- [ ] TBD — run /paul:plan when Phase 4 is complete
|
||||
Harden the Hub's side of the on-prem polling protocol:
|
||||
- Credit deduction when on-prem reports delivered jobs
|
||||
- Failed job reporting (on-prem couldn't send → Hub handles retry)
|
||||
- Feature flags returned with every poll
|
||||
- Suspension flag in config response
|
||||
- Graceful degradation: stale `processing` jobs reclaimed back to `pending` if on-prem goes offline
|
||||
|
||||
## Plan
|
||||
|
||||
### 5-01: sync/poll enhancements + credit deduction (Hub side)
|
||||
|
||||
**Changes to `backend/app/routers/sync.py`:**
|
||||
- Accept `job_failed_ids: list[str]` in poll body — increment retry_count, mark failed at 5
|
||||
- Deduct 1 SMS credit per job in `report_sent_ids`, write SmsCreditLedger entries
|
||||
- Fire `send_low_credit_alert` if credits fall below threshold after deductions
|
||||
- Add `feature_flags` dict to config response (tier-based, reusing `_tier_features()` from licenses.py)
|
||||
- Add `suspended: bool` to config response
|
||||
- Return `403` with `reason` field if license is expired (not just revoked)
|
||||
|
||||
**Changes to `backend/app/models/sms.py`:**
|
||||
- Add `delivered_via: Mapped[str | None]` column (`"pull"` | `"push"` | None)
|
||||
|
||||
**Changes to `backend/app/tasks/sms.py`:**
|
||||
- Add `reclaim_stale_jobs()` task — reset `processing` jobs older than 5 minutes back to `pending`
|
||||
(handles on-prem going offline mid-cycle)
|
||||
- Mark jobs sent by Celery push path as `delivered_via = "push"`
|
||||
|
||||
**Changes to `backend/app/worker.py`:**
|
||||
- Schedule `sms.reclaim_stale_jobs` every 5 minutes
|
||||
|
||||
### 5-02: Alembic migration for delivered_via column
|
||||
|
||||
Add `delivered_via VARCHAR(10)` nullable to `sms_jobs` table.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
On-prem TapTrack (every 30s):
|
||||
POST /api/sync/poll
|
||||
Body: {
|
||||
license_key: "TTUB-XXXXX",
|
||||
report_sent_ids: ["uuid1", "uuid2"], ← jobs on-prem successfully sent
|
||||
job_failed_ids: ["uuid3"] ← jobs on-prem could NOT send
|
||||
}
|
||||
|
||||
Hub response:
|
||||
{
|
||||
sms_jobs: [...], ← up to 50 pending jobs to send
|
||||
config: {
|
||||
sms_sender_name: "...",
|
||||
sms_credits: 47.0,
|
||||
school_status: "active",
|
||||
suspended: false,
|
||||
feature_flags: { sms: true, reports: true, ... }
|
||||
}
|
||||
}
|
||||
|
||||
Credit flow (pull path):
|
||||
On-prem sends → reports sent_ids next poll → Hub deducts 1 credit per job
|
||||
(NOT deducted when job is dispatched — only when confirmed sent)
|
||||
|
||||
Graceful degradation:
|
||||
sms.reclaim_stale_jobs (every 5min):
|
||||
UPDATE sms_jobs SET status='pending'
|
||||
WHERE status='processing' AND updated_at < now() - 5min
|
||||
→ If on-prem dies mid-poll, jobs return to Celery push queue
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user