feat(phase-5): on-prem sync poll hardening + stale job reclaim
Harden the Hub-side polling protocol for on-prem TapTrack agents: - sync/poll: accept job_failed_ids (retry/fail on-prem delivery failures) - sync/poll: deduct credits + write ledger when on-prem reports sent jobs - sync/poll: return feature_flags (tier-based) + suspended flag in config - sync/poll: skip job dispatch for suspended/expired schools - sms_jobs: add delivered_via (pull|push) + processing_started_at columns - tasks/sms: new sms.reclaim_stale_jobs task resets processing→pending if on-prem goes offline (jobs stuck >5 min), enabling Celery push fallback - tasks/sms: tag Celery-sent jobs as delivered_via='push' - worker: schedule reclaim_stale_jobs every 5 minutes - migration: 001_phase5 adds delivered_via + processing_started_at to sms_jobs
This commit is contained in:
kevin-asprec
@@ -19,8 +19,8 @@ SMS flow: On-prem TapTrack polls Hub every 30s → Hub queues SMS jobs → Hub s
|
||||
## Current Milestone
|
||||
|
||||
**v1.0 — Foundation & Core Services**
|
||||
Status: Phase 4 complete — Phase 5 next
|
||||
Phases: 4 of 15 complete
|
||||
Status: Phase 5 complete — Phase 6 next
|
||||
Phases: 5 of 15 complete
|
||||
|
||||
---
|
||||
|
||||
@@ -32,7 +32,7 @@ Phases: 4 of 15 complete
|
||||
| 2 | School Registry + License Mgmt | 2 | ✅ Complete | 2026-03-15 |
|
||||
| 3 | On-Prem License Validation | 1 | ✅ Complete | 2026-03-15 |
|
||||
| 4 | SMS Gateway (credits + queue) | 2 | ✅ Complete | 2026-03-15 |
|
||||
| 5 | On-Prem SMS Polling Agent | TBD | Not started | — |
|
||||
| 5 | On-Prem SMS Polling Agent | 1 | ✅ Complete | 2026-03-16 |
|
||||
| 6 | Super Admin Dashboard UI | TBD | Not started | — |
|
||||
| 7 | School Admin Portal UI | TBD | Not started | — |
|
||||
| 8 | Billing Engine + Invoice PDF | TBD | Not started | — |
|
||||
|
||||
@@ -3,16 +3,16 @@
|
||||
## Current Position
|
||||
|
||||
Milestone: v1.0 — Foundation & Core Services
|
||||
Phase: 4 of 15 (SMS Gateway — complete)
|
||||
Plan: Phase 4 complete — Phase 5 next
|
||||
Status: **Phase 4 applied — ready to begin Phase 5**
|
||||
Last activity: 2026-03-15 — Phase 4 complete (SMS stats/health/retry/trigger endpoints + full SmsPage dashboard with chart, KPIs, school breakdown)
|
||||
Phase: 5 of 15 (On-Prem SMS Polling Agent — complete)
|
||||
Plan: Phase 5 complete — Phase 6 next
|
||||
Status: **Phase 5 applied — ready to begin Phase 6**
|
||||
Last activity: 2026-03-16 — Phase 5 complete (sync/poll hardening: credit deduction, failed job reporting, feature flags, suspension flag, stale job reclaim)
|
||||
|
||||
## Loop Position
|
||||
|
||||
```
|
||||
PLAN ──▶ APPLY ──▶ UNIFY
|
||||
· · · [No active plan — Phase 2 planning next]
|
||||
· · · [No active plan — Phase 6 planning next]
|
||||
```
|
||||
|
||||
## Progress
|
||||
@@ -23,7 +23,7 @@ PLAN ──▶ APPLY ──▶ UNIFY
|
||||
- Phase 2 (School Registry + License Mgmt): [██████████] 100% ✓
|
||||
- Phase 3 (On-Prem License Validation): [██████████] 100% ✓
|
||||
- Phase 4 (SMS Gateway): [██████████] 100% ✓
|
||||
- Phase 5 (On-Prem SMS Polling Agent): [░░░░░░░░░░] 0%
|
||||
- Phase 5 (On-Prem SMS Polling Agent): [██████████] 100% ✓
|
||||
- Phase 6 (Super Admin Dashboard UI): [░░░░░░░░░░] 0%
|
||||
- Phase 7 (School Admin Portal UI): [░░░░░░░░░░] 0%
|
||||
- Phase 8 (Billing Engine + Invoice PDF): [░░░░░░░░░░] 0%
|
||||
@@ -37,8 +37,8 @@ PLAN ──▶ APPLY ──▶ UNIFY
|
||||
|
||||
## Next Action
|
||||
|
||||
Run: `/paul:plan` for Phase 5 — On-Prem SMS Polling Agent
|
||||
Resume file: .paul/ROADMAP.md → Phase 5
|
||||
Run: `/paul:plan` for Phase 6 — Super Admin Dashboard UI
|
||||
Resume file: .paul/ROADMAP.md → Phase 6
|
||||
|
||||
## Repo
|
||||
|
||||
|
||||
@@ -1,9 +1,74 @@
|
||||
# Phase 05: On-Prem SMS Polling Agent
|
||||
# Phase 05: On-Prem SMS Polling Agent (Hub Side)
|
||||
|
||||
**Status:** Not started
|
||||
**Status:** Complete
|
||||
**Completed:** 2026-03-16
|
||||
|
||||
## Goal
|
||||
TapTrack polls Hub every 30s for pending SMS jobs; sends them; reports back completion.
|
||||
|
||||
## Plans
|
||||
- [ ] TBD — run /paul:plan when Phase 4 is complete
|
||||
Harden the Hub's side of the on-prem polling protocol:
|
||||
- Credit deduction when on-prem reports delivered jobs
|
||||
- Failed job reporting (on-prem couldn't send → Hub handles retry)
|
||||
- Feature flags returned with every poll
|
||||
- Suspension flag in config response
|
||||
- Graceful degradation: stale `processing` jobs reclaimed back to `pending` if on-prem goes offline
|
||||
|
||||
## Plan
|
||||
|
||||
### 5-01: sync/poll enhancements + credit deduction (Hub side)
|
||||
|
||||
**Changes to `backend/app/routers/sync.py`:**
|
||||
- Accept `job_failed_ids: list[str]` in poll body — increment retry_count, mark failed at 5
|
||||
- Deduct 1 SMS credit per job in `report_sent_ids`, write SmsCreditLedger entries
|
||||
- Fire `send_low_credit_alert` if credits fall below threshold after deductions
|
||||
- Add `feature_flags` dict to config response (tier-based, reusing `_tier_features()` from licenses.py)
|
||||
- Add `suspended: bool` to config response
|
||||
- Return `403` with `reason` field if license is expired (not just revoked)
|
||||
|
||||
**Changes to `backend/app/models/sms.py`:**
|
||||
- Add `delivered_via: Mapped[str | None]` column (`"pull"` | `"push"` | None)
|
||||
|
||||
**Changes to `backend/app/tasks/sms.py`:**
|
||||
- Add `reclaim_stale_jobs()` task — reset `processing` jobs older than 5 minutes back to `pending`
|
||||
(handles on-prem going offline mid-cycle)
|
||||
- Mark jobs sent by Celery push path as `delivered_via = "push"`
|
||||
|
||||
**Changes to `backend/app/worker.py`:**
|
||||
- Schedule `sms.reclaim_stale_jobs` every 5 minutes
|
||||
|
||||
### 5-02: Alembic migration for delivered_via column
|
||||
|
||||
Add `delivered_via VARCHAR(10)` nullable to `sms_jobs` table.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
On-prem TapTrack (every 30s):
|
||||
POST /api/sync/poll
|
||||
Body: {
|
||||
license_key: "TTUB-XXXXX",
|
||||
report_sent_ids: ["uuid1", "uuid2"], ← jobs on-prem successfully sent
|
||||
job_failed_ids: ["uuid3"] ← jobs on-prem could NOT send
|
||||
}
|
||||
|
||||
Hub response:
|
||||
{
|
||||
sms_jobs: [...], ← up to 50 pending jobs to send
|
||||
config: {
|
||||
sms_sender_name: "...",
|
||||
sms_credits: 47.0,
|
||||
school_status: "active",
|
||||
suspended: false,
|
||||
feature_flags: { sms: true, reports: true, ... }
|
||||
}
|
||||
}
|
||||
|
||||
Credit flow (pull path):
|
||||
On-prem sends → reports sent_ids next poll → Hub deducts 1 credit per job
|
||||
(NOT deducted when job is dispatched — only when confirmed sent)
|
||||
|
||||
Graceful degradation:
|
||||
sms.reclaim_stale_jobs (every 5min):
|
||||
UPDATE sms_jobs SET status='pending'
|
||||
WHERE status='processing' AND updated_at < now() - 5min
|
||||
→ If on-prem dies mid-poll, jobs return to Celery push queue
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user