Bemnet Beshah

Projects / Research

KineticAugment

Labeled sign-language video is scarce, so I am building a way to generate extra training clips that keep each sign’s meaning. I convert MediaPipe landmarks to 3-D SMPL-X (body), MANO (hands) and a fitted head, perturb them within linguistically grounded limits, and convert them back. The conversion is measured and the first augmentation trials exist. The test that matters, a recognizer trained with and without the generated clips, has not been run.

Problem

Generic pose augmentation can silently change a sign. Moving a hand from chin to forehead, or a straight path to an arc, makes a different word. The linguistics sources I found give no numeric tolerances, so limits have to be set from evidence and checked per clip.

The input is noisy too. MediaPipe’s skeleton is not rigid: bone lengths change 2 to 15% from frame to frame within a clip.

My role

I work on a fork of the imics-lab/KineticAugment repository. The upstream framework (landmark-based augmentation, YAML profiles, documentation) already existed in the imics-lab repository. Starting 1 October 2026 I built the body, hand and face round trips, the measurement scripts, the findings documents, the augmentation evidence review, the first trials and a round-trip web viewer. That work was merged into upstream main on 19 October 2026.

How it works

sign video (WLASL)
   |
MediaPipe: pose-world, hand and face landmarks
   |
   +-- body : 6 arm landmarks --analytic swing IK--> SMPL-X pose + per-clip shape
   +-- hands: 21 landmarks per hand ------fit------> MANO / SMPL-X hand pose
   +-- face : 478 points -> head, neck, jaw; signed hand-to-face distances
   |
augment once per clip (yaw, mirror, rate, scale, shift, small arm angles)
   |
back to landmarks, then checks: location, contact, handshape unchanged?
   |
accepted clips join the training split only   -->  recognizer (not yet run)

Body error is the median distance between a landmark and its round-trip reconstruction, as a percentage of shoulder width. At a typical 0.33 m shoulder width, 1% is about 3.3 mm. Hand error is in millimetres from the wrist.

Key decisions

The gate is a rigid-skeleton floor, not bit-exactness. A bit-exact round trip is impossible because the detector is non-rigid. MediaPipe’s jitter was the alternative threshold, but a constant offset has zero jitter. The gate became the best error any constant-length skeleton reaches on the same clip (2.36% on the pilot).

Bone directions stay exact. HybrIK-style adaptive IK lowered position error from 3.52% to 3.43% but bent arm bones by a median 1.21 degrees. Augmentation perturbs joint angles, so I kept directions exact and gave up 0.09 points.

I dropped body-shape augmentation. The idea was to vary SMPL-X shape and re-solve the arms. In a trial it moved wrists 68 to 83 mm. Across 25 clips, signer differences in MediaPipe’s arm proportions are at the noise floor, and 24 of 25 fitted shapes sat on the plausibility bound. I removed it on 18 October 2026.

Augmentation ranges come from evidence, and the split protects the test. Tiers come from sign-linguistics sources and published ablations (SPOTER, Spoken2Sign). Original videos are split before augmentation, each generated clip stays with its source in training, and validation and test use untouched originals.

What was hard

The starting converter was not a conversion. The repo’s SMPL-X converter stored bone displacements in the rotation slots and added them back, so its 3.7e-9 error was arithmetic. I then found one MediaPipe instance reused across clips leaked tracking state, shifting landmarks up to 114 mm, and re-extracted everything.

Error depends on pose, not only on the converter. One ranking number mixed three causes: body proportions SMPL-X cannot take, pose-dependent error, and detector noise. On clip 63229 the error is 1 to 2% with the arm down and 9 to 10% with the hand at the chin.

Hands never reach the head in the fitted scene. On 143 hand-near-face frames across 20 clips, hands sit a median 0.18 m from the head along the camera axis, and none came within 0.137 m. The gap is already in MediaPipe’s pose-world output before fitting, so contact is unmeasured.

Results

All figures are from WLASL footage, dated 5 to 11 October 2026, and none measures recognition.

  • Body round trip: 3.52% direct and 3.99% held out on 5 pilot clips. On 20 random unseen clips (1,236 frames, seed 20261007) it was 3.13% and 3.37%.
  • Hands (MANO, in-sample): 7.38 mm mean on the pilot, missing the 5 mm target. On the random 20, 5.10 mm mean, 2.53 mm tips and 14.59 mm p95, so the mean misses by 0.1 mm. The hand was detected in 42% of side-frames.
  • Face geometry: head rotation agrees with FaceLandmarker to a median 0.57 to 0.58 degrees. Both come from the same landmarks, so this is not independent accuracy. Jaw rank correlation has median 0.92 on the 8 of 20 clips that could be scored.
  • Face stream integration: 25 clips and 1,646 of 1,646 frames passed through the face-stream classifier code with finite training updates. The labels were synthetic and no recognition accuracy was measured.
  • First augmentation trials (clips 17083, 57641): scale and off-centre shifts leave the shoulder-normalised pose unchanged. Arm-joint changes were three to four times too strong, moving wrists 94 to 115 mm, and I am reworking them.

Limitations and next steps

The decisive experiment is open: one recognizer, fixed preprocessing and training budget, trained on originals versus originals plus accepted augmentations, scored on untouched original validation and test clips. The face integration run had no official annotation data available.

Other limits: only WLASL footage is tested, some clips share signers, and the full-figure hands are about 12% too large because body shape is fitted from arm lengths alone. The location, contact and handshape checks are not built yet. Next: build them, obtain ASL-LEX labels for per-sign gating, and run one-augmentation-at-a-time recognizer ablations.