Kubecon Keynote: Platform Alchemy: Transforming Kubernetes Into Generative AI Gold

In this KubeCon Japan keynote presentation, Alexa Griffith from Bloomberg and Mauricio "Salaboy" Salatino from Diagrid discuss the evolution of platform engineering from traditional applications to generative AI (GenAI), emphasizing the continuous adaptation and transformation of existing tools rather than discarding them. The speakers highlight three core pillars for building effective platforms: X-as-a-Service, Developer Experience, and Observability.

Key Takeaways:

• X-as-a-Service: The presentation shows how to expose complex underlying tools as simple, consumable services for developers and data scientists, such as "database as a service" or "LLM as a service," by leveraging tools like Crossplane, Dapr, Quasar, and Envoy AI Gateway. This shifts the focus from managing deployments to optimizing access patterns.
• Developer Experience: The speakers stress the importance of enabling developers to build complex applications effectively, even with intricate toolchains. They advocate for providing production-tested frameworks and highlight projects like Dapr Agents for building agentic programming models.
• Observability for GenAI: The talk emphasizes the new demands for observability in GenAI, including tracking tokens, measuring token-level latency, and tracing prompts through agents. These new metrics are crucial for enterprise-ready LLMs and need to be integrated into Service Level Objectives (SLOs).

Alexa and Mauricio illustrate how traditional Kubernetes platforms, while robust for standard applications (e.g., scaling web apps, managing relational databases), require significant adaptations for predictive and generative AI. This includes specialized serving runtimes, new hardware like GPUs, and the need for vector databases. They introduce tools like KServe for operationalizing GenAI deployments and address challenges such as model caching, latency tuning, and distributed inference for large models.

The conclusion: the power of modern platforms lies in the composability and interoperability of their components. By focusing on open protocols and open-source solutions, organizations can create an "elixir" that transforms raw infrastructure into a golden GenAI platform, fostering innovation and accelerating development for platform teams.

‍