> For the complete documentation index, see [llms.txt](https://docs.themspkb.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.themspkb.com/ai-for-msps/foundational-ai-concepts/bert.md). # BERT ### Summary **BERT** (Bidirectional Encoder Representations from Transformers) is a 2018 model from Google that learns language by reading text in **both directions at once** — considering the words before *and* after a given word — rather than strictly left to right. It popularized the two-step recipe of [Pretraining and Fine-Tuning](/ai-for-msps/foundational-ai-concepts/pretraining-and-fine-tuning.md): train one general model on massive unlabeled text, then adapt small copies of it to specific tasks. BERT dominated language-*understanding* benchmarks and still powers a great deal of behind-the-scenes AI — search relevance, classification, intent detection — even though the more visible chat assistants come from the rival GPT lineage of [Large Language Models](/ai-for-msps/foundational-ai-concepts/large-language-models.md). ### In plain terms BERT is an **encoder-only** [Transformer Architecture](/ai-for-msps/foundational-ai-concepts/transformer.md): it is built to *read and represent* text, not to generate it. Its training used two clever tricks: * **Masked Language Modeling (MLM).** During training, random words are hidden and the model must guess them from the surrounding context on *both* sides. Fill-in-the-blank forces genuine two-directional understanding — this is BERT's signature idea. * **Next Sentence Prediction (NSP).** The model is shown two sentences and learns to judge whether the second naturally follows the first — teaching it relationships between sentences. The paper released two sizes: **BERT-Base** (\~110 million parameters) and **BERT-Large** (\~340 million). Both were pretrained on English Wikipedia and a large book corpus, then fine-tuned to set new records across a suite of language-understanding tasks (the GLUE benchmark, question answering, and more). ### Why it matters for MSPs * **Not all "AI" is a chatbot.** Much of the AI quietly embedded in MSP tools — routing a ticket to the right queue, tagging a document, detecting the intent of an email, ranking KB search results — is a classification job that a BERT-style model does cheaply and reliably. Recognizing this helps you tell *workhorse* AI from *generative* AI when [evaluating where it shows up](/ai-for-msps/ai-in-the-msp-stack/where-it-shows-up.md). * **Fine-tuning means your data may train the model.** BERT's paradigm involves adapting a model on task-specific data. When a vendor offers a model "tuned on your tickets," that is fine-tuning — which makes the **no-training guarantees** and data-handling questions on the [Data Handling & Privacy](/ai-for-msps/ai-security/data-handling-and-privacy.md) and [What It Can't Do Yet](/ai-for-msps/ai-in-the-msp-stack/what-it-cant-do-yet.md) pages directly relevant. * **Understanding ≠ generating.** BERT reads well but does not write fluent long-form answers. Knowing which kind of model sits behind a feature sets correct expectations for clients. ### Related concepts [Transformer Architecture](/ai-for-msps/foundational-ai-concepts/transformer.md) · [Attention Mechanism](/ai-for-msps/foundational-ai-concepts/attention-mechanism.md) · [Pretraining and Fine-Tuning](/ai-for-msps/foundational-ai-concepts/pretraining-and-fine-tuning.md) · [Large Language Models](/ai-for-msps/foundational-ai-concepts/large-language-models.md) · [Foundation Models](/ai-for-msps/foundational-ai-concepts/foundation-models.md) **In the MSP KB:** [Data Handling & Privacy](/ai-for-msps/ai-security/data-handling-and-privacy.md) · [Where It Shows Up](/ai-for-msps/ai-in-the-msp-stack/where-it-shows-up.md) ### Contradictions & debates * **Bidirectional (BERT) vs. one-directional (GPT).** BERT's central argument is that reading text left-to-right *only* is a real limitation, and that bidirectional context is needed for strong understanding — a point the paper makes explicitly against earlier GPT-style models. The [Large Language Models](/ai-for-msps/foundational-ai-concepts/large-language-models.md) lineage never adopted bidirectionality and still reached strong, sometimes state-of-the-art results, showing the limitation was surmountable by scale and generation-first design. Both approaches remain in use for different jobs; the "contradiction" resolved into a **division of labor**, not a winner. * **Fine-tuning vs. in-context learning.** BERT assumes you *fine-tune* a separate model per task. Two years later, [GPT-3](/ai-for-msps/foundational-ai-concepts/large-language-models.md) argued that fine-tuning has real downsides (it needs labeled data and can latch onto spurious patterns) and demonstrated learning tasks from a few prompt examples with no weight changes — see [In-Context Learning](/ai-for-msps/foundational-ai-concepts/in-context-learning.md). This is the sharpest methodological disagreement among the five papers. ### Source paper Devlin et al. (2018), *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding* — *** **Key terms**: *BERT*, *bidirectional*, *encoder-only*, *masked language modeling*, *next sentence prediction*, *GLUE benchmark*. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.themspkb.com/ai-for-msps/foundational-ai-concepts/bert.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.