# MyChild Engine — Autonomous Validation Study

> 71,939 synthetic profiles. Cohen's kappa 0.906, sensitivity 91.4%, specificity 98.8%. 14 engine improvements across 4 autonomous research runs. Every number is reproducible. This is a software verification study, NOT clinical validation — ground-truth labels came from an AI clinical evaluator, not licensed clinicians.

- Authors: Harsh Songra & Aafreen Ansari · Last updated: July 2026
- Clinical review: not yet — actively seeking clinical partners
- Canonical HTML page: https://www.mychildapp.in/study

## Headline metrics

| Metric | Value | Scope |
|---|---|---|
| Cohen's Kappa | 0.906 | Combined set |
| Sensitivity | 91.4% | Flags real concerns |
| Specificity | 98.8% | Doesn't cry wolf |
| Observations | 34,587 | Domain-profile pairs scored |

## Two-phase validation

**Phase 1 — Curated set (hand-verified):** 1,747 profiles, 12,991 observations, kappa 1.000, 100% sensitivity & specificity. Profiles promoted from the adversarial set after manual review confirmed unambiguous agreement. Kappa = 1.000 is expected by construction; this validates logic on clear-cut cases, not edge-case performance.

**Phase 2 — Adversarial stress test:** 70,192 profiles, 21,596 observations, kappa 0.852, sensitivity 86.5%, specificity 98.0%. Profiles designed to break the engine: regression patterns, sparse data, contradictory answers, preterm edge cases, single-domain flags. Of the 781 disagreements, most were under-flags — the engine is cautious.

## What "adversarial" means (18 categories)

Regression (10,578), Sparse data (9,072), Single domain (8,786), Mutations (6,500), Exhaustive age (5,616), Contradictory (5,208), Clear delay (5,073), Clear typical (4,964), Preterm (4,752), Borderline (2,897), Combinatorial (2,544), Real-world (1,914), and more.

**315 critical findings need human review** — adversarial profiles where the engine and the AI evaluator disagreed on clinically meaningful classifications. Logged and triaged, not hidden.

## Comparison with clinical tools

| Tool | Sensitivity | Specificity | Validation | Sample |
|---|---|---|---|---|
| ASQ-3 | 82–97% | 83–93% | Clinical | ~18,000 |
| M-CHAT-R/F | 85–95% | 93–99% | Clinical | — |
| PEDS | 74–79% | 70–80% | Clinical | ~1,500 |
| MyChild (adversarial) | 86.5% | 98.0% | Synthetic | 21,596 obs |
| MyChild (combined) | 91.4% | 98.8% | Synthetic | 34,587 obs |

ASQ-3, M-CHAT-R/F, and PEDS are clinically validated with real patient data. Ours are synthetic. You can't put them on equal footing.

## 14 improvements across 4 runs

- **Run 1:** isolated single-flag downgrade, cross-domain sparse correction, regression bypasses evidence gate, cross-domain regression protection
- **Run 2:** soft regression detection, multi-regression escalation, caregiver uncertainty detection, developmental sequence anomaly, asymmetric screening thresholds, + 5 more (RF scoring, label normalization, evidence gate tuning, low_concern handling, sparse precaution)
- **Run 3:** 8 refinement commits polishing edge cases
- **Run 4:** zero changes — 325 threshold experiments all returned "keep current parameters"

## Threshold optimization

The autoresearch system swept every tunable parameter (delta-not-yet interactions, delta-unsure values, infant and toddler grace periods, yellow/orange/red thresholds) — 325 experiments. Every single one came back "keep current parameters." Current values: T_yellow = 2, T_orange = 4, T_red = 6, grace 3w/12w. To improve from here we need to change algorithms, not knobs.

## RBSK alignment

RBSK screens 270 million children across India, independently validated at 97% sensitivity and 96.4% specificity against ASQ-3. Our question bank is mapped domain-by-domain against RBSK screening items. That doesn't make us RBSK-validated. See https://www.mychildapp.in/rbsk.md

## The original verification

Before the autoresearch system: 108 hand-labeled scenarios, 294 labeled observations, 98.6% agreement, Cohen's kappa 0.971, 4 threshold cases, 45 abstentions. Per-domain: Gross Motor (72 obs, 100%), Cognitive/Play (22, 100%), Self-Help (14, 100%), Vision/Hearing (23, 100%), Expressive Language (67, 98.5%), Social-Emotional (45, 97.8%), Receptive Language (34, 97.1%), Fine Motor (17, 94.1%). Every miss was a borderline over-flag — when the engine gets it wrong, it errs toward concern.

## The honest version

**What this shows:** kappa held at 0.852 under adversarial attack; 98.0% adversarial specificity; 14 algorithmic fixes shipped from autoresearch findings; 325 threshold experiments all said "keep"; question bank maps to CDC 2022 milestones and RBSK items; fully open source and reproducible.

**What this does NOT prove:** clinical validation with real patient data (needs IRB oversight and clinical partners); real-world sensitivity/specificity; cross-language/cross-culture performance; clinical-grade thresholds. 315 critical adversarial findings still need a clinician's review. A prospective clinical study is next.

## Reproduce it yourself

```bash
npm install mychild-engine
npx mychild-engine validate --profiles data/synthetic-profiles.json --format markdown
npx mychild-engine autoresearch --runs 4 --adversarial 70000
```

Research: https://github.com/hsongra11/mychild-app/tree/main/docs/research · npm: https://www.npmjs.com/package/mychild-engine

## Sources

1. Zubler JM, et al. *Pediatrics* 2022;149(3) — doi:10.1542/peds.2021-052138 (CDC 2022 revised milestones, 75th percentile standard)
2. Lipkin PH, Macias MM. *Pediatrics* 2020;145(1) — doi:10.1542/peds.2019-3449 (AAP surveillance & screening policy)
3. CDC "Learn the Signs. Act Early." — cdc.gov/act-early/milestones
4. RBSK — rbsk.mohfw.gov.in
5. Synthea validation study, *BMC Med Inform Decis Mak* 2019 — doi:10.1186/s12911-019-0793-0
6. "Synthetic data can aid the analysis of clinical outcomes," *PNAS* 2024 — doi:10.1073/pnas.2414310121
7. "Synthetic Validation of Pediatric Trust Instruments Using LLMs," MedRxiv 2025

---

Pages: [Home](/index.md) · [About](/about.md) · [Study](/study.md) · [Study Data](/study-data.md) · [Docs](/docs.md) · [RBSK](/rbsk.md) · [Privacy](/privacy.md) · [Terms](/terms.md) · [Contact](/contact.md)