Question 1

What is Mixture of Experts (MoE) architecture in LLMs?

Accepted Answer

Mixture of Experts (MoE) is an architecture that replaces the dense feed-forward layer in a standard Transformer with a sparse layer containing multiple 'expert' sub-networks and a learned routing mechanism. For each input token, the router selects only k experts (typically 2-4) from a total of N experts (typically 8-128) to process that token. This means the model has a large total parameter count — Kimi uses 236 billion total parameters — but only activates a fraction per token (the 'active parameters'), dramatically reducing compute per forward pass compared to a dense model of equivalent total size. The result is frontier-class capability at 3-5x lower inference cost compared to a dense model with the same total parameters.

Question 2

How does Kimi achieve a 1 million token context window?

Accepted Answer

Kimi's 1 million token context window is enabled by several technical innovations beyond the MoE architecture: sparse attention mechanisms that compute attention over a compressed representation of distant context rather than attending to every token; memory-efficient KV cache management that uses quantisation and selective eviction to keep the KV cache for 1M tokens within GPU memory bounds; and a training curriculum that progressively extended context length during pre-training to develop the model's ability to use long-range dependencies effectively. The 1M context allows Kimi to process entire codebases, legal document sets, or multi-book corpora in a single context — use cases that required chunking and retrieval with shorter-context models.

Question 3

What is Kimi k1.5 and how does it compare to other reasoning models?

Accepted Answer

Kimi k1.5, released in January 2025, is Moonshot AI's reasoning model trained with reinforcement learning to perform extended chain-of-thought deliberation before producing answers — the same paradigm as OpenAI o1. Kimi k1.5's key innovation is long chain-of-thought RL training with a 128k context window for the thinking trace, enabling it to work through longer reasoning chains than o1's initial implementation. On the AIME 2024 mathematical competition benchmark, Kimi k1.5 scored comparably to o1, demonstrating that Chinese AI labs had independently developed and matched Western reasoning model capabilities. k1.5 is significant evidence of the convergence of the global AI research frontier.

Question 4

What are the enterprise use cases for Kimi's 1 million token context?

Accepted Answer

The 1 million token context unlocks use cases previously requiring fragmented retrieval approaches: entire codebase analysis and refactoring (a large enterprise codebase of 1-2M tokens fits in a single context); legal due diligence (complete contract sets, regulatory filings, and correspondence for M&A transactions); financial analysis (full annual reports, earnings call transcripts, and analyst reports for multiple years across multiple companies); medical record synthesis (complete patient history including all notes, test results, and imaging reports); and book-length document authoring (maintaining narrative and factual consistency across 200,000+ word documents). The key advantage is eliminating retrieval pipeline complexity — rather than chunking and RAG, the entire document corpus is processed in a single inference call.

Kimi & Moonshot AI — Mixture of Experts Architecture & Long-Context LLMs

What This Guide Covers

Mixture of Experts — How Kimi Achieves Frontier Performance at Lower Cost

Kimi k1.5 — Reasoning Model with RL Training

Topics Covered in This Guide

Read the Full Guide + Download Free Sample

Frequently Asked Questions

Brief Summary

Extended Summary

Related Guides in the SimuPro Knowledge Store

SimuPro Data Solutions — Cloud Data Engineering & AI Consultancy