Core Function: Enable application and user access to large language models (LLM). Makes an LLM available for real-world use after it has been trained. Manages the LLM, loading it into memory, and allocating resources to ensure it runs smoothly.
Inference vs. Training: "Inference" is the process of generating output (like text) from a prompt, while "training" is the process of creating the model.
Hardware: Typically runs on specialized, powerful hardware like GPUs to handle the computational requirements of LLMs.
Efficiency: Uses techniques like batching (grouping requests) and quantization (reducing model size) to process requests faster and more efficiently.
API Interface: Provides an API (Application Programming Interface) that allows other software to send requests to the model and receive responses.
Scalability: Designed to handle many requests simultaneously.