Blog Overview
RAG answers from your approved documents with clear citations, but only after careful chunking and Arabic evaluation. This guide explains when to use it, when not to, and an eight-item pre-launch checklist.
Support teams answer the same Arabic questions every day: where is my order, what does this fee mean, when does this channel open. The answers sit in manuals, policy files, and email archives. Rules-based chatbots handle narrow flows, but they break when wording changes, when a customer writes in dialect, or when an updated policy needs a new branch before the next release.
The practical test is simple: do not launch RAG on the strength of one demo. Launch it on a set of real Arabic questions, reviewed by Arabic-speaking reviewers. This article explains when RAG beats chatbots, when it should not be used, and how to build the evaluation before the first customer message.
When RAG beats rules and chatbots
RAG wins when the knowledge base is long and changes often: fees, opening hours, eligibility rules, document steps. A rules-based bot needs a rule for each phrasing and a task for each policy update. RAG reads the same approved document the agent reads, so updating the document once changes the next answer. The biggest gain is on long-tail questions: rare questions asked in many ways, where writing a rule for each wording is impractical.
Rules remain better for some work: a narrow, well-known flow such as a single status lookup or one activation step, where the steps must be deterministic and auditable. The useful design combines both: rules execute the action and RAG explains the policy. That is the pattern Rasad follows in AI solutions, where the line is quote only: scope sets the price and the timeline. Rasad has delivered 45+ projects, including 3 government entities, over 5+ years, and each started from one clear task.
When RAG should not be used
Do not use RAG for payments, identity, or legal advice. A system that charges an amount, confirms a person's identity, or decides a contractual obligation needs deterministic checks, human approval, and an audit trail. RAG may retrieve the right passage and still draft a confident answer around an old or incomplete paragraph. A citation does not fix a wrong payment decision. It should also not close an account, change contractual data, or rule in a dispute. In these cases its first job is to shorten the search, then hand over to a person who owns the decision.
The workable routing is: RAG answers informational questions, rules execute after clear confirmation, and a person handles exceptions, complaints, and high-value cases. A message that asks for an action should show a review screen with clear numbers before executing, and the system should record who approved and when and on what wording. If RAG crosses that boundary, it stops being help and becomes risk. A safe build blocks execution by design and keeps execution in testable code.
Arabic specifics you cannot skip
Documents are in Modern Standard Arabic, but customers do not only write that way. Support messages mix Gulf, Levantine, and Egyptian dialect with Modern Standard Arabic in the same thread, plus short colloquial forms without diacritics. Retrieval has to match meaning across registers, not word overlap alone. Terms such as fees and subscription may mean the same thing or differ by service. The honest test is to use real chats in their original forms, including spelling slips and dialect, then observe what is retrieved and what is missed.
Code-switching between Arabic and English is the norm: product names, error codes, and order numbers may appear in Latin letters inside an Arabic sentence. An English identifier should stay intact, and bidi isolation should keep brackets and punctuation from flipping. Numerals are a product decision: Western digits in financial screens or Arabic-Indic in editorial content, but one published rule. Dates may be Gregorian or Hijri, and diacritics are often missing in customer writing. This is where RTL failures appear, which we review in RTL design mistakes.
| Request type | First path | Human role |
|---|---|---|
| Order-status or fee question | RAG from policy and help content | Steps in if information is missing or conflicts |
| Refund or payment action | Rules with confirmation before execution, RAG to explain only | Approves and reviews |
| Identity or account change | Rules with verified identity checks | Verifies and documents |
| Legal or contractual question | RAG summarises the source with a clear notice | Reviews the final wording |
| Unclear or sensitive request | RAG refuses and hands over to a person | Owns the case from the start |
Pipeline decisions that decide quality
Chunking is the first decision. Split by headings and sections, not by word count alone. Keep a table whole, keep the condition with its outcome, and keep the Hijri or Gregorian date with the amount it belongs to. Each chunk should carry the document title, date, and language, so the system can cite the right source and the operator can spot an old document before a customer does. Scanned documents need good Arabic OCR, because one misread digit or date poisons retrieval.
Embedding choice and evaluation are two sides of one task. Choose a multilingual embedding model that supports Arabic, but compare it on your own questions rather than on vendor claims. Every factual answer should cite the document title, the section, and a link the user can open, and cite only the retrieved chunks. If there is no source, there should be no answer. Hide sourceless replies from the interface and log them in a review queue so the operator knows where knowledge is missing.
From reading to shipping
Want this built, not just explained?
Bring us the problem behind the article. We will tell you what it takes, what it costs, and whether we are the right team.
Hallucination controls
Refusal, handoff, and threshold work together. If retrieved chunks are weak or conflicting, the system should refuse in polite Arabic, offer a handoff to an agent, and log the question as a knowledge gap. The threshold that separates retrieve and answer from refuse and hand over is a value tuned on the evaluation set and revisited with real chats each week. Never answer an amount or an obligation from weak retrieval, even when the reply sounds confident.
Generation controls complete the set. Prompts should require answering from chunks only, keep wording consistent, block adding fees or dates absent from the source, and preserve English codes as they are. Log the query, chunks, and answer for each case, so any reply can be checked against its source within minutes. The conversation should stay in the language the customer wrote in, because a formal Modern Standard reply to a dialect question can read as an answer to a different thing.
PDPL and support transcripts
Support transcripts are personal data: names, phone numbers, identity numbers, account numbers, and order numbers. Minimise what the system collects, hide what you do not need, and block indexing of raw uncontrolled text. Real chats are suitable for evaluation and error review only under limited access, with a defined retention window and a clear deletion path. The PDPL Implementing Regulation PDF is the formal reference here, but the final legal wording needs specialist review.
The practical workflow: redact phone and identity numbers before any evaluation or logging, keep the development set separate from the test set, and do not send raw transcripts to an external provider without reviewing where the data is sent and stored. The Personal Data Protection Law has been fully enforceable since September 2024. SDAIA published draft amendments to the Implementing Regulation on 6 October 2026 and the consultation closes 5 November 2026. The draft proposes storing personal data inside the Kingdom and notifying breaches within 72 hours of awareness, but it remains draft until adopted.
Evaluation before launch
Build the evaluation set from real chats, not invented questions the model will like. Include dialect variants, mixed Arabic and English questions, questions the documents cannot answer, and dangerous questions that ask for payment, identity, or an obligation. For each question, name the expected source document and whether refusal or handoff is expected. Keep the tuning set separate from the test set, so the threshold is not tuned on the same questions it is judged on. Reviewers must be Arabic speakers who read the dialect the question was written in, not machine translation.
The measures to track are retrieval quality, whether the answer follows the chunks, citation correctness, refusal correctness, the share of harmful replies under human Arabic review, handoff correctness, and response time inside the chat. Use human review for the hard sample, automated checks for missing citations, and rerun the full set after any change in documents, chunking, or prompts. Log every failure in its original Arabic wording. That log is the product evidence.
Before you launch
A frozen evaluation set. Real questions in dialect and mixed numerals, with expected sources and Arabic review.
A citation audit. Every factual answer linked to its document and section, and no sourceless reply shown to the customer.
Refusal and handoff tests. Questions without a source return a polite Arabic refusal and offer an immediate handoff to an agent.
Dialect and code-switching tests. Four dialects, short forms, and Latin names inside Arabic sentences, with codes left intact.
Numbers and dates tests. Amounts, ranges, and Hijri and Gregorian dates read in the right order on every screen.
Redaction and retention. Phone and identity numbers hidden before evaluation, with retention, deletion, and access documented.
Full logging per case. Query, chunks, and answer traceable, with a human review queue for regulated topics.
A small pilot. One queue with human review before replying, then expansion in batches based on the failure log.
Frequently asked questions
When is a rules-based bot better than RAG? When the flow is one known path and the steps must be deterministic. Rules are faster and easier to audit. RAG is better when knowledge is long and changing.
How do we handle a mixed Arabic and English question? Keep the Latin identifier intact, isolate bidi direction, and test the full sentence on a real Arabic screen before launch.
What does RAG cost? AI solutions are quote only: scope sets the price and the timeline. Start with one task and verified questions.
How do we know answers are correct without an invented number? Through human Arabic review and tying each answer to its source. Retrieval, handoff, and timing are honest operating numbers, while an accuracy figure without a set is a pitch.
Can real chats be used for evaluation? Yes, after redacting personal data and limiting them to internal review, with a short retention window and restricted access.
Sources
- PDPL Implementing Regulation — the official reference for processing and breach duties.
- SDAIA Personal Data Protection portal — fully enforceable since September 2024.
- Material 3: bidirectionality and RTL — layout reverses alignment and order, and components are designed for both directions.
- Google Fonts: Noto Kufi Arabic — an open Arabic interface face with clear licensing.
This article is engineering guidance, not legal advice. The final legal wording for support-data handling needs review by a data-protection specialist.
