Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
AI Sovereignty and Local LLM Deployment
- Addressing the risks associated with cloud LLMs, including data retention, input training, and external jurisdictional concerns.
- Understanding the Ollama architecture, comprising the model server, registry, and OpenAI-compatible API.
- Comparing Ollama with alternative solutions such as vLLM, llama.cpp, and Text Generation Inference.
- Reviewing model licensing terms for Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Setting up Ollama on Linux with support for CUDA and ROCm.
- Configuring CPU-only fallbacks and optimizing for AVX/AVX2 instruction sets.
- Implementing Docker deployments and managing persistent volume mappings.
- Structuring multi-GPU environments and defining VRAM allocation strategies.
Model Management
- Retrieving models from the Ollama registry using commands like 'ollama pull llama3.'
- Importing GGUF formatted models from HuggingFace and TheBloke.
- Evaluating the tradeoffs of different quantization levels, such as Q4_K_M, Q5_K_M, and Q8_0.
- Managing model switching and understanding limits on concurrent model loading.
Custom Modelfiles
- Mastering Modelfile syntax, including directives for FROM, PARAMETER, SYSTEM, and TEMPLATE.
- Fine-tuning parameters such as temperature, top_p, and repeat_penalty.
- Engineering system prompts to define role-specific behaviors.
- Creating and publishing custom models to a local registry.
API Integration
- Leveraging the OpenAI-compatible /v1/chat/completions endpoint.
- Implementing streaming responses and utilizing JSON mode.
- Integrating with frameworks like LangChain, LlamaIndex, and custom applications.
- Managing authentication and rate limiting using reverse proxies.
Performance Optimization
- Optimizing context window sizing and managing KV cache efficiency.
- Handling batch inference and parallel request processing.
- Allocating CPU threads and maintaining NUMA awareness for optimal performance.
- Monitoring GPU utilization and addressing memory pressure issues.
Security and Compliance
- Implementing network isolation for secure model serving endpoints.
- Establishing pipelines for input filtering and output moderation.
- Performing audit logging of prompts and generated completions.
- Verifying model provenance and hash integrity.
Requirements
- Proficiency in intermediate-level Linux administration and container management.
- A conceptual understanding of machine learning and transformer architectures.
- Working knowledge of REST APIs and JSON data structures.
Target Audience
- AI engineers and developers looking to migrate away from cloud LLM APIs.
- Organizations with strict data sensitivity requirements that preclude the use of cloud-hosted models.
- Government and defense sectors requiring air-gapped, secure language model solutions.
