“Why They Leave”: Synthesis of 1,500 Public Remittance App Reviews
Multilingual extraction and structured coding (ES, EN, PT, SV) without compromising corporate confidentiality. Evaluated with inter-annotator agreement (Cohen's Kappa) and mathematical sizing linking qualitative pain directly to the technical roadmap.
PROCESSED CORPUS
1,500
4 apps • 4 languages
HUMAN-AI AGREEMENT
κ = 0.81
Cohen's Kappa (n=225)
TIME SAVINGS / SPRINT
~34 h
From 40h manual to 6h
TOP DETECTED PAIN
34.0%
Delivery delay
01. Problem Statement & Public Extraction
The critical inflection point: a migrant sends money to their family and the transaction gets stuck 'in transit'. They reach out to customer support while simultaneously leaving a 1-star app review. Without systematic quantification, product teams prioritize recent anecdotes rather than solving the root cause of trust erosion.
Compliance & Confidentiality Guarantee:
Corpus gathered entirely from public Apple App Store RSS feeds and Google Play public APIs across direct competitors (Remitly, Wise, WorldRemit, and international calling apps). Zero exposure of internal corporate data.
02. Coding Methodology & Human-Model Validation
LLMs only achieve reliable qualitative synthesis when guided by formal taxonomies. The codebook started in v1 with 8 categories and evolved to v2 (11 categories) after auditing classification edge cases.
FACTUAL / EXPLICIT CODES
κ = 0.86 (High)
TRANS_DELAY, APP_TECH, REFUND_DELAY
INTERPRETATIVE / SUBTEXT CODES
κ = 0.61 (Moderate)
SUPPORT_NO_REPLY, FRAUD_SUSP
Anti-hallucination guardrail: no theme enters the final report without at least 3 linked verbatim customer quotes.
03. Opportunity Sizing: From Reviews to Business Impact
Extrapolated across a baseline of 500,000 monthly transactions, a 3% support contact rate (15,000 tickets), and a $10 direct cost per human touchpoint:
| Code | Corpus (n=1,500) | % Mix | Est. Volume | Support Cost | Opportunity Score |
|---|---|---|---|---|---|
| TRANS_DELAY (Delivery delay) | 510 | 34.0% | 5,100 tickets/mo | $51,000/mo | 7,650 |
| SUPPORT_NO_REPLY (Support loop) | 315 | 21.0% | 3,150 tickets/mo | $31,500/mo | 4,410 |
| KYC_BLOCK (Verification) | 210 | 14.0% | 2,100 tickets/mo | $21,000/mo | 2,520 |
| PAYOUT_FAIL (Payout failure) | 165 | 11.0% | 1,650 tickets/mo | $16,500/mo | 2,145 |
| REFUND_DELAY (Withheld refund) | 120 | 8.0% | 1,200 tickets/mo | $12,000/mo | 1,920 |
Visible arithmetic: Opportunity Score = Monthly incidence × Churn severity multiplier. Pain points #1 and #2 account for 55% of all friction and inform the Case 2 prototype.
04. Inference Failure Gallery & Architecture Pivot
Failure: Sarcasm undetected in lightweight model
“Incredible service, 6 days waiting for my mother's medicine to arrive.”
The cost-effective classifier assigned OTHER_UNCAT (compliment) due to surface lexical positivity.
Correction: System prompt instructions were updated to prioritize underlying factual delay over superficial positive adjectives.
Architecture Pivot: Vector Embedding Clustering
Initial attempt: Unsupervised clustering using vector embeddings to identify emergent topics.
Failure & Correction: The vector space clustered reviews by language (Portuguese together, Swedish together) rather than by problem taxonomy. Reverted to a structured codebook evaluated by a multilingual LLM.
Product Chain: Discovery → Build
55% of complaints concentrated in transaction status and unresponsive support justified building the agent.