The decision is not whether a model can write Arabic, but what it may know, what it may do, and who takes over when it fails. This article separates resolution from deflection, retrieval from fine-tuning, the evaluation set, and when buying beats building.
The first demo of an AI support assistant is always convincing: a tidy question, a short answer, and a response time that never tires. Then a real question arrives. A customer writes in dialect, attaches a photo of a receipt, asks to change an order that has already shipped, and wants an answer before the store closes. That is where the difference appears between an assistant that demonstrates a model and one that works inside the system, knowing what it may say and what it may do.
The decision is not whether a model can write Arabic. It can. The decision is what the assistant is allowed to know, what it is allowed to do, who takes over when it fails, and how you will prove that it improved rather than appeared to improve. Those questions decide whether this becomes a product or a pilot that ends when the budget does.
Deflection is a vanity metric
Deflection counts the conversations that did not reach a human. It does not count whether the customer solved the problem. An assistant can deflect a conversation in three ways: it refuses the request, it loops, or it exhausts the customer until they close the window. In all three the number looks good in the report, and the customer is worse off than when they started.
Five measures are worth building on: whether the issue was resolved without reopening, how long resolution took from the customer's side, whether escalation reached the right team with context, whether the customer returned with the same question, and how much human time the assistant saved. These do not improve through the model alone. They improve through knowledge, integration, and escalation policy.
The only defensible target is a comparison: the same queue, the same period, the assistant on and then off, with human review of a sample of conversations. That design stops you from comparing a quiet summer with a launch month, and from claiming an improvement that happened because the human team changed at the same time.
Arabic is a language problem and a dialect problem
Modern Standard Arabic is understood, but people do not always write in it. Dialects differ between regions, and the same customer moves between standard Arabic and dialect inside one conversation. Some customers spell words as they sound, with different letters each time. A model that performs on a demo paragraph can fail on a real ticket because the spelling is not stable.
Add the interface layer: right-to-left text, numerals, Hijri and Gregorian dates, addresses, and mixed names. A conversation carrying a number, a postal code and an address can break visually even when the answer is correct. Add voice if the assistant handles calls: dialect, background noise and automatic correction compound their errors before the text ever reaches the model.
So do not build the evaluation set from clean questions written by the product team. Build it from real text: tickets, messaging conversations, emails and call notes, in a distribution that reflects your actual channels. Label it by intent, dialect and clarity, and mark the cases that need account data, the cases that need an exception, and the cases that must reach a human.
Retrieval or fine-tuning
Retrieval-augmented generation gives the model passages from your documents at answer time, so it builds the answer on existing text and can point to it. Fine-tuning changes the model's behaviour and style through examples; it is a poor store for changing facts. Confusing the two is the first cause of an assistant that answers confidently about a policy that no longer exists.
| Question | Retrieval | Fine-tuning |
|---|---|---|
| What changes | The model's access to your documents at answer time | The model's behaviour and style through training examples |
| Freshness | Updates when the index updates | New facts require retraining |
| Evidence | Can point to the passage it used | Cannot cite a source reliably |
| Style and format | Set by instructions and examples in the prompt | Learned from examples, more consistent |
| Cost of change | Re-index a document | Rebuild a dataset and retrain |
| Failure mode | Retrieves the wrong passage or misses one | Invents facts confidently and is hard to trace |
| Best use | Policies, product facts, and procedures that change | Classification, extraction, and consistent style |
| Data needed | Documents and a retrieval pipeline | Labelled examples of good answers |
The working rule: start with retrieval for any question whose answer lives in a document, a policy, or an order state. Use fine-tuning when you need stable classification, field extraction from text, or a style that does not drift between writers. Many working systems use both: retrieval for knowledge, fine-tuning for answer shape and intent classification.
What it may know and what it may do
An assistant that can read everything is not a good assistant; it is a risk with permissions. Start from the minimum: which data does it need for this class of question, at what level of access, and for how long. Order status needs an identifier and verification. An address change, cancellation or refund is a change of financial state, and needs stronger verification, an approval policy, and a log.
Separate reading from writing in the design, not in intent. Route access through scoped interfaces rather than an account that sees the whole database. Prevent one customer's data from appearing in another customer's conversation, and test it explicitly: prompt injection, an attempt to read an order by number, and a file upload carrying hidden instructions. These tests are not theoretical; they are the part that fails first in demos.
A support transcript is personal data. It may carry a name, an identity number, an address, a purchase history, and a photo of a document. It needs retention, redaction of sensitive fields, access control, and a contract with every provider that touches it. The draft amendments published in October 2026 propose breach notification within 72 hours; an assistant that keeps everything forever makes that obligation harder, not easier.
The evaluation set is the product
From reading to shipping
Want this built, not just explained?
Bring us the problem behind the article. We will tell you what it takes, what it costs, and whether we are the right team.
An assistant does not improve because the model changed. It improves because you hold a set of cases where you know the right answer, and measure every change to prompts, index or integration against it. Without that set, every improvement is an opinion, and every regression is discovered through a customer complaint a week later.
Build the set in layers: questions answerable from a document, questions that need account data, exception cases, complaints, out-of-scope questions, mixed dialects, suspicious requests, and ambiguous cases. For each case, score whether it is correct, grounded in a source, safe, escalated to the right place, and phrased appropriately. Review a sample by hand and record disagreements between reviewers, because the disagreement itself exposes an unclear policy.
Run the set on every change: a new prompt, an updated document, a new provider, a new integration. Keep a frozen copy you do not tune against, or you end up training the assistant on its own test. The set is a company asset, like the codebase; whoever owns it can change providers without losing the ability to judge the result.
Buy, build, or wait
Buying fits a standard job: repeated questions, order status, transfer to a team, known channels, and a shallow integration. The team gets value quickly and does not have to build monitoring and evaluation from scratch. The risks are a vendor's limited data model, knowledge that is hard to export at exit, and a region or subprocessor list that does not match what your client requires.
Building fits when knowledge and workflow are the advantage: proprietary policies, complex products, deep integration with orders and CRM, Arabic as a first-class requirement, and a need to control the model, the data and the evaluation. You own the experience, and you also own the responsibility: uptime, security, cost, human review, and whoever answers when the assistant gets it wrong.
Waiting is also a project decision. Wait when you have no clean knowledge, no access to real transcripts, no policy owner, and no way to measure the outcome. An assistant built on contradictory knowledge will deliver the contradiction to the customer faster. Waiting here is not hesitation; it avoids the cost of a pilot nobody can judge.
The pragmatic middle is to buy the channel and interface, and build the retrieval, tools and escalation policy. You avoid building a chat window from scratch without depending on a vendor that does not understand your order system. Decide the exit from the start: what knowledge, evaluation data and transcripts remain with you if you change providers a year from now?
What moves the cost
The model is rarely the largest line. The large lines are preparing and owning the knowledge, integrating with orders and CRM, building the evaluation set, human review, and monitoring after launch. Add the security and privacy review, because every permission you give the assistant is a new attack surface.
Knowledge is not one file. It is return policy, delivery windows, exceptions, shipping rates, order states, and the language of each channel. It needs an owner who updates it on every change, or the assistant becomes an outdated reference that answers with confidence. Integration needs testing against real cases: a cancelled order, a partial order, a customer without an email, and an address outside the delivery area.
That is why we recommend starting with one journey, one channel and one evaluation set. An assistant that covers everything in the first release hides its errors in the variety. One that covers a single journey well gives you evidence you can defend to management, and a base to expand. See Rasad's published bands for how discovery and build are priced, and do not assume a fixed figure before the journey is defined.
The decision checklist before you buy or build
Use this list before choosing a vendor or writing a line of code. If any line has no clear answer, the project has not started yet.
Define resolution and escalation before choosing a tool: what counts as done, and when a customer must reach a human.
Real transcripts from every channel and dialect, not demo questions written by the product team.
An owner for knowledge who updates policies, prices and procedures on every change.
A permission map for what the assistant reads and does, at the lowest level, per question type.
A written handoff contract covering the context passed with the conversation, the queue, the language, and what was tried.
A frozen evaluation set with clear criteria, human review, and an update cycle that leaves the reference copy intact.
A policy for what it may do, not only what it may say: cancellation, refunds, and address changes.
A privacy and security review before the pilot, with transcript redaction, retention, and provider contracts.
Frequently asked questions
Will the assistant replace the support team? Not in the shape most teams imagine. It changes the mix: repeated questions handled automatically, complex cases reaching a human with better context. The team that disappears is usually the one that never had a clear escalation policy.
Does it understand Saudi dialects? Some of them, and you should not assume full understanding. Test it on your own transcripts, measure how many cases it understood, then decide which dialects to support and which to route to a person.
Do we start with retrieval or fine-tuning? Start with retrieval when the answers live in documents, policies or order state. Use fine-tuning for classification, extraction and stable style. Many systems need both, but retrieval first gives you faster updates and clearer evidence.
How do we measure success? Compare the same queue before and after: resolved without reopening, correct escalation, no repeat question, human time saved, with human review of a sample. Do not start from a deflection number.
Is customer data safe? It is safe when permissions are scoped, transcripts are redacted, retention is defined, contracts are clear, and the security review happens before launch. The draft data protection amendments propose breach notification within 72 hours, which makes logging and retention part of the design rather than an appendix.
Sources
- Rasad: AI solutions — how we define the journey and build evaluation and integration.
- Rasad: retrieval for Arabic customer support — architecture, knowledge, and evaluation.
- SDAIA: Personal Data Protection Law portal — the text in force and the guidance material.
- Rasad: the PDPL checklist for client portals — permissions, retention, and logs.
- Rasad's published price bands — how discovery and build stages are priced.
Cover image: MÁV Zrt., CC BY-SA 3.0, via Wikimedia Commons.

