> For the complete documentation index, see [llms.txt](https://docs.themspkb.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.themspkb.com/ai-for-msps/foundational-ai-concepts/transformer.md). # Transformer Architecture ### Summary The **Transformer** is the neural-network design introduced in the 2017 paper *Attention Is All You Need*. It builds an entire model out of the [Attention Mechanism](/ai-for-msps/foundational-ai-concepts/attention-mechanism.md) and simple feed-forward layers, discarding the step-by-step recurrence that earlier language models relied on. Because it processes all words in parallel, it can be trained on enormous datasets efficiently — which is what made today's [Large Language Models](/ai-for-msps/foundational-ai-concepts/large-language-models.md) possible. The "GPT" in ChatGPT stands for **Generative Pre-trained Transformer**, and Microsoft Copilot, [BERT](/ai-for-msps/foundational-ai-concepts/bert.md), and nearly every current AI feature in the MSP stack are Transformers under the hood. ### In plain terms Earlier models read a sentence like a person reading aloud — one word at a time, carrying a running memory forward. That was slow and tended to "forget" the start of a long passage by the time it reached the end. The Transformer does something different. It looks at the whole input at once and uses [attention](/ai-for-msps/foundational-ai-concepts/attention-mechanism.md) to let every word directly consult every other word. Key pieces of the design: * **Encoder and decoder stacks.** The original design had two halves — an *encoder* that reads and represents the input, and a *decoder* that generates output. The base model stacked six of each. * **Multi-head attention.** Several attention operations run in parallel, each tracking a different kind of relationship in the text. * **Positional encoding.** Because the model reads all words at once (with no inherent sense of order), it adds a signal marking each word's position, so "server restarted the service" isn't confused with "service restarted the server." * **Feed-forward layers and residual connections.** Standard neural-network plumbing that processes and stabilizes the attention output. Later models often keep only one half: [BERT](/ai-for-msps/foundational-ai-concepts/bert.md) is **encoder-only** (built to *understand* text), while GPT-style models are **decoder-only** (built to *generate* text). ### Why it matters for MSPs * **It is the thing behind "the AI."** When a PSA/RMM vendor, a documentation tool, or a security product advertises AI, it is almost certainly a Transformer. Knowing the one architecture demystifies the whole category. * **It explains the cost and speed trade-offs.** Transformers are expensive to run because attention compares everything to everything. That cost is why AI features carry per-seat or per-token pricing, and why large inputs are slower and pricier — relevant when you model [implementation and ROI](/ai-for-msps/ai-in-the-msp-stack/where-it-shows-up/implementation-and-roi.md). * **It explains why the same core tech shows up everywhere.** A single architecture powering search, chat, code, and security tooling is exactly the *homogenization* the [Foundation Models](/ai-for-msps/foundational-ai-concepts/foundation-models.md) page describes — a convenience that also concentrates risk. ### Related concepts [Attention Mechanism](/ai-for-msps/foundational-ai-concepts/attention-mechanism.md) · [BERT](/ai-for-msps/foundational-ai-concepts/bert.md) · [Large Language Models](/ai-for-msps/foundational-ai-concepts/large-language-models.md) · [Foundation Models](/ai-for-msps/foundational-ai-concepts/foundation-models.md) · [Pretraining and Fine-Tuning](/ai-for-msps/foundational-ai-concepts/pretraining-and-fine-tuning.md) **In the MSP KB:** [AI vs. Automation](/ai-for-msps/ai-in-the-msp-stack/ai-vs.-automation.md) · [AI in the MSP Stack](/ai-for-msps/ai-in-the-msp-stack.md) ### Contradictions & debates * The Transformer paper's core claim — that **recurrence is unnecessary** — was a direct challenge to the prevailing RNN/LSTM approach of its time. History sided with the Transformer; none of the other papers in this section dispute it. * A genuine fork appears *downstream* of the Transformer, not within it: the [BERT](/ai-for-msps/foundational-ai-concepts/bert.md) lineage and the [Large Language Models](/ai-for-msps/foundational-ai-concepts/large-language-models.md) lineage take the same architecture in incompatible directions — bidirectional-and-fine-tuned versus left-to-right-and-prompted. See those pages for the disagreement. ### Source paper Vaswani et al. (2017), *Attention Is All You Need* — *** **Key terms**: *Transformer*, *encoder/decoder*, *decoder-only*, *positional encoding*, *multi-head attention*, *GPT (Generative Pre-trained Transformer)*. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.themspkb.com/ai-for-msps/foundational-ai-concepts/transformer.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.