In this talk, we will explore the field of Explainable AI (XAI), with a particular focus on techniques used to gain insights into Large Language Models (LLMs). We will show how XAI can be applied to LLMs, specifically analyzing attention maps and the Transformer architecture. Additionally, we will walk through the process of building an LLM from scratch, using visualization tools to delve into its inner workings. The talk will address key questions surrounding the explainability of LLMs, such as: How can we interpret these vast models? Do LLMs primarily memorize or generalize? What are the emergent capabilities of LLMs, and how do they arise?