Cantonese Data Services for Pangeanic Cross-Lingual RAG

The Client Need

Cross-lingual Retrieval-Augmented Generation (RAG) systems can fail when the query, retrieved document, and answer language do not align. Cantonese adds complexity because written Traditional Chinese, colloquial Cantonese expressions, and English business terminology often appear in the same workflow.

What NLPConsultancy Supplied

We mapped the semantic relationships across language variants to support accurate retrieval:

• Parallel document, sentence, and query-answer pairs for retrieval and answer evaluation.
• Cantonese, Traditional Chinese, and English variants where project requirements needed them.
• Query paraphrase sets, terminology maps, and metadata for domain, source, and language variant.

Dataset Characteristics

The deliverables were structured as JSONL, TSV, or CSV. Each pair was specifically engineered to stress-test embedding models, retrieval benchmarks, and RAG evaluation logic under cross-lingual conditions.

Quality & Validation Process

Human validation was deployed to reduce alignment noise, eliminate duplication, and remove hallucination-prone pairs that often plague automated data scraping.

Outcome for Pangeanic

• Pangeanic received reusable evaluation assets for retrieval and answer quality across languages.
• The material helped test whether systems worked for Cantonese queries, not just English prompts.
• Structured benchmark data supported the direct comparison of embedding models, rerankers, and RAG pipelines.
• Controlled data preparation significantly reduced dependence on unaudited, noisy bilingual web material.