There is no single best Arabic language model to pick from a leaderboard. Falcon, ALLaM, Jais and Fanar are families with several releases, and the one that fits depends on your dialects, tasks, deployment constraints and license terms. The comparison worth trusting is one you run yourself: pin exact checkpoints, test them on your own Arabic tasks under comparable conditions, and read the results by slice. This guide explains how to design that evaluation.
It deliberately names no winner. Whether to run an open-weight model on your own infrastructure or call a hosted multilingual model is a separate operational decision, covered in our guide to local versus hosted models for Arabic.
A family name is not something you can test. Each family publishes multiple models over time, and they differ in size, training, context length and license. Start a candidate sheet with one row per exact checkpoint, recording its name, release date, license, context length, supported languages and how you would actually run it.
The primary sources show why this matters:
- Falcon. The Technology Innovation Institute (TII) announced Falcon Arabic on 21 May 2025. Check the license and specifications of the exact checkpoint you would deploy rather than relying on the announcement.
- ALLaM. The model card for ALLaM-7B-Instruct-preview, published on HUMAIN's Hugging Face page, describes a 7-billion-parameter model with a 4,096-token context, Arabic and English support and an Apache-2.0 license, developed by the National Center for AI at SDAIA. That describes one preview checkpoint, not the whole ALLaM family.
- Jais. MBZUAI announced Jais 2 together with Inception and Cerebras, so the older Jais-13B is no longer the only Jais option to consider.
- Fanar. Fanar has published release notes for version 2.0 of its Arabic-oriented stack. Confirm access terms and model details directly before including it.
Treat each announcement as a developer's description of its own model, not as a neutral ranking.
Figure 1 — Pin each candidate checkpoint. Record license, context and hosting.
Models do not win in general; they win at tasks. Write down the three to five tasks the system must perform, each with an example input and a description of a good output. Typical Arabic workloads include answering questions from internal policy documents, summarizing customer complaints, drafting formal correspondence, classifying support tickets and extracting fields from invoices or forms.
Each task needs its own scoring rules. A summary is judged on coverage and faithfulness, a classification on whether the label is right, and an answer from documents on whether its claims are supported. If retrieval is involved, give every model the same retrieved passages so that you compare the models rather than the search; our enterprise RAG guide explains how to evaluate that pipeline.
The test set is where Arabic evaluation succeeds or fails. Have native speakers write cases, rather than translating an English set, and tag each case with the features that make Arabic hard:
- Variety and dialect: Modern Standard Arabic, Gulf, Egyptian, Levantine or Maghrebi, as your users actually write them.
- Code-switching: sentences that mix Arabic and English, and Arabic written in Latin characters.
- Spelling variation: hamza forms, taa marbuta and alef maqsura written in different ways, with and without diacritics.
- Domain terms: product names, legal or medical vocabulary and the abbreviations your staff use.
- Formats: Eastern and Western Arabic numerals, Hijri and Gregorian dates, and right-to-left output that contains Latin text.
Use real but de-identified inputs where you can. Our guide to evaluating LLM hallucinations covers labeling rules and how to think about the size of each slice.
A controlled comparison keeps task conditions stable while changing the model. Keep the instructions, retrieved context, output length limits and decoding settings the same wherever the models allow it, and use each model's documented chat format so that formatting errors are not mistaken for weak Arabic. Record the exact checkpoint, serving setup and date of every run. Tokenizers and decoding options differ, so record unavoidable differences and use comparable output budgets rather than assuming identical parameter values mean identical behavior. Repeat sampled generations and report variation; a small score gap on a small test set may not be meaningful.
Context length becomes a hard constraint here, not a score. If a task needs long documents in the prompt, a checkpoint with a 4,096-token context, such as the ALLaM preview model above, cannot take the same input without changes to chunking. Record that as a fit issue and, if needed, test a version of the task that every candidate can run.
Where your data constraints permit, include a hosted multilingual model as a baseline. It helps test whether an Arabic-focused model offers an advantage for your tasks; if hosted access is disallowed, compare against a permitted multilingual checkpoint.
Figure 2 — Keep conditions comparable. Review outputs without model names.
Use native-speaker reviewers who do not know which model produced each output, and give them a rubric with separate judgments for correctness, faithfulness to the provided sources, fluency and register, terminology and appropriate declining. Register matters: a reply in casual dialect may be wrong for a formal letter, while stiff Modern Standard Arabic may be wrong for a chat reply.
Report results per slice, with case counts, so you can see, for example, whether one checkpoint handles Gulf dialect questions well but struggles with code-switched ones. Read the failing outputs themselves. Patterns such as misread negation, wrong gender agreement or invented terminology tell you more than a single rate. If a model grades the answers, validate it against Arabic human labels first.
Quality results answer only part of the question. Record the operational facts beside them rather than inside one blended score: whether the license permits your commercial use, whether you can host the checkpoint where your data must stay, expected latency and serving cost on your infrastructure, and how often the model is updated. A model that scores well but cannot be deployed under your constraints is not a candidate. Keep a short decision record listing the checkpoint chosen, the evidence, the constraints and the date, so the decision can be revisited when new releases appear.
Rankings age quickly. New checkpoints appear, licenses and access terms change, and results depend on the task and the test set. A table of scores from someone else's evaluation would tell you little about your users, which is why this article describes a method instead. When a family releases a new model, add it as a new row and re-run the same evaluation.
This method is TecnoStack's guidance, not a requirement of any model developer. If you are choosing an Arabic-capable model for a production system, our AI systems service can help you design and run the comparison.
- Technology Innovation Institute. Falcon Arabic launch announcement, 21 May 2025.
- HUMAIN. ALLaM-7B-Instruct-preview model card: parameters, context length, languages and license for this checkpoint.
- Mohamed bin Zayed University of Artificial Intelligence. Jais 2 release announcement, with Inception and Cerebras.
- Fanar. Fanar 2.0 release notes.
