Clinical CV Overhaul — Detailed Changes (2026-09-17)¶

Summary¶

A rework of the clinical video assessment feature across OktaPT-API (19 files) and Okta-Mobile (123 files, including a new native module) landed in early September 2026: structured test types with five shoulder movements and a ten-step shoulder protocol, a recording setup screen, on-device live pose tracking with three engines, a hold-still countdown, per-clip review and retake with background uploads, a server-side quality gate, a resumable and BUSY-aware analysis pipeline, image-plane range-of-motion measurement, and in-app video playback with a synced overlay.

Cross-repo PRs: OktaPT-API #413 (clinical-cv-overhaul) and #415 (clinical-cv-upgrades); Okta-Mobile #61, #62 and #64. No feature flag gates the change; capability gating is by device support (livePoseSupport.ts) and by derived_metrics.schema_version on the server. Mobile version bumped to 2.1.0 for both tenant brands; native builds are required (OTA cannot carry the native module or the minSdkVersion change).

For the engineering reference see ../internal/clinical-video-assessment.md. For the user-facing description see ../clinical-assessment.md.

User-facing changes¶

Structured tests: testType / side / movement instead of free text; shoulder ROM offers abduction, flexion, extension, external rotation and internal rotation; All shoulder tests runs both sides of all five, ordered by camera position, with per-step skip and retake.
New Setup screen (framing rule, camera position, start position, MovementDemo stick figure) between test selection and the camera.
Live overlay while recording: full skeleton for non-ROM tests; tested limb with goniometer arc, straight-line guide, live current/peak and framing chips for ROM tests.
Start test arms a hold-still countdown (3 s, joints visible and still) instead of recording immediately; a plain timer on devices without a pose engine.
Per-clip review card (peak, tracking, length, poor-tracking warning) with Retake; uploads run in the background and are only confirmed at Finish & send.
Quality rejection: unusable clips come back as "Video needs a retake" with a reason and a retake button; Recent Tests shows an amber Retake badge and sorts newest first.
Results: measured-ROM card per side, video-quality and AI-confidence badges, in-app playback with a synced skeleton overlay and a peak jump, peak frame as the hero still with the rest behind "All frames"; the cross-validation section was removed from the screen.
Settings gear: engine choice (MediaPipe / Apple Vision / YOLO11) and a Motion Cap try-out screen.
~150 new strings in both locales/en.json and locales/ru.json.

Data model (OktaPT-API)¶

New enums ClinicalTestType (GAIT, SHOULDER_ROM, KNEE_ROM, SQUAT, SINGLE_LEG_BALANCE, POSTURE, CUSTOM), ClinicalTestSide (LEFT, RIGHT, BILATERAL), ClinicalTestMovement (FLEXION, ABDUCTION, EXTENSION, EXTERNAL_ROTATION, INTERNAL_ROTATION); ClinicalAssessmentStatus gains REJECTED_QUALITY.
New ClinicalAssessment columns: testType, testSide, testMovement, testFreeText, metricsSummary Json?, liveMetrics Json?, rejectionReasonCode String?.
Migrations: 20260903211626_clinical_assessment_test_type_and_quality, 20260915213336_added_test_test, 20260916223015_added_new_tests.

OktaPT-API¶

File	Change
`src/routes/clinical-assessment/controllers.ts` (+214)	zod-validated create body with casing tolerance and ROM-only `movement`; `REJECTED_QUALITY` accepted by upload-url (fresh key) and upload-complete; `liveMetrics` on upload-complete; new fields and `statusDetail` on get/list; new `getPlayback`; `peak_for` on overlay frames; `serializeStatus` shim mapping `REJECTED_QUALITY` → `FAILED` for older app builds; chained direct pipeline runs when Redis is absent
`src/routes/clinical-assessment/index.ts`	`GET /:id/playback`
`src/services/assessmentPipelineService.ts` (+668/−256)	Resumable `processAssessment(id, { resume })`; 503 BUSY polling up to `CLINICAL_VIDEO_BUSY_WAIT_MS`; error classification (422 quality → `REJECTED_QUALITY`, other 4xx unrecoverable, 5xx/timeouts retryable); `CLINICAL_VIDEO_TIMEOUT_MS`; request carries `test_type`, `side`, `movement`, `test_description`
`src/services/clinicalAssessmentPrompts.ts` (new)	Three prompts extracted; Pass 2 branches on `schema_version` (v1 verbatim, v2 with test declaration, conventions, trust weighting, ROM rules); Pass 3 quality ceiling and authoritative measured summary; `PASS2_MAX_METRICS_CHARS`
`src/services/clinicalTestTypeMap.ts` (new)	Prisma enum ↔ Python wire values; `ROM_TEST_TYPES`
`src/services/liveRomMetrics.ts` (new)	zod schema and `parseUploadCompleteBody` (32 KB, 400 samples)
`src/services/clinicalMetricsSummary.ts` (new)	Whitelisted `metricsSummary` extraction (10 KB cap)
`src/worker.ts`	Passes `resume: job.attemptsMade > 0`
`lib/openai_response_formats/clinical_biomechanical_analysis_schema.ts`, `clinical_final_analysis_schema.ts`	Tightened field descriptions
`src/services/__tests__/assessmentPipelineBusy.test.ts`, `clinicalAssessmentPrompts.test.ts`, `clinicalMetricsSummary.test.ts`, `clinicalTestTypeMap.test.ts`, `liveRomMetrics.test.ts` (new)	Pipeline BUSY handling, prompt branching, summary extraction, enum mapping, live-metrics validation

Okta-Mobile¶

Screens (app/(doctor)/(clinical-assessment)/): _layout.tsx (5 → 7 screens), index.tsx (structured tests, movement modal, protocol row, gear), setup.tsx (new), record.tsx (rewritten on VisionCamera V5: useVideoOutput 720p / 4 Mbps persistent recorder + frame output, steps[] runs, review layer, expo-file-system/legacy background uploads, deferred upload-complete), motion-cap.tsx (new), processing.tsx (quality-rejection branch and retake routing), results.tsx (playback, measured-ROM card, badges; frame gallery and cross-validation removed), recent-tests.tsx (retake badge, newest-first).

Components (components/clinical/): LivePoseOverlay, FullSkeletonOverlay, PlaybackPoseOverlay, AssessmentVideo, MeasuredRomCard, RecordingReviewCard, ClipPreview, StepChecklist, CountdownOverlay, MovementDemo, LiveOverlaySettingsSheet.

Hooks: useLivePose (native landmarker + frame output, worklet thread → One-Euro smoothing → Reanimated shared values), useLiveRom (measure, peak, chips, capture, arming), useLiveEngine (stored engine choice).

Pure logic (lib/clinical/, unit-tested in lib/__tests__/): jointAngles, angleUnwrap, liveRomConfig, liveRomSmoothing, liveRomMetrics, startPosition, lowFpsGuard, oneEuro, landmarkTransform, poseGeometry, skeletonPaths, poseConnections, playbackOverlay, testSteps, liveEngineSetting, livePoseSupport.

Service: services/clinicalAssessmentService.ts — typed enums and metrics, getPlayback(), uploadComplete(id, liveMetrics?), normalizeTestFields.

Native module: modules/pose-landmarker/ — Nitro module (src/PoseLandmarker.nitro.ts, ios/HybridPoseLandmarker.swift, android/.../HybridPoseLandmarker.kt, generated nitrogen/) wrapping MediaPipe Tasks Vision (iOS + Android), Apple Vision and YOLO11n-pose CoreML; synchronous VIDEO-mode process(frame), warmUp(), delegate reporting; bundled pose_landmarker_heavy.task (both platforms) and yolo11n_pose.mlpackage (iOS).

Build/config: app.config.js (version 2.1.0 both tenants, minSdkVersion: 24, plugins/withPrivateReactHeaders.js), package.json (react-native-vision-camera 5.2.3, react-native-worklets 5.2.3, react-native-nitro-modules 0.37.1, react-native-nitro-image 0.15.2, @okta/pose-landmarker), docs/live-rom-overlay.md (engineering log).

Compatibility notes¶

Older app builds receive status: "FAILED" for quality-rejected assessments (statusDetail carries the truth), so they keep working without the retake flow.
Runs analysed before the server change have derived_metrics v1: no metricsSummary, no measured-ROM card, old Pass 2 prompt, and video playback without the overlay.
The Python service and the API enum map (clinicalTestTypeMap.ts ↔ app/models/schemas.py) must be deployed together when test types change.

Verification¶

API: five new jest suites plus the existing pipeline tests.
Mobile: lib/__tests__ suites for every pure module (angles pinned to the same fixtures as the Python angles.py), npx tsc --noEmit, and the device QA matrix in docs/live-rom-overlay.md (iPhone 12 / 15 Pro, Pixel 6a, Samsung A54; simulator = disabled path).