ARTICLE AD BOX
I am currently working on a Python based Gen AI project that requires the efficient deployment and serving of multiple LLMs specifically models with different parameter counts ( Llama-2 7B and Mistral 7B) on a single GPU infrastructure to minimize latency and maximize throughput.
I am utilizing the Hugging Face transformers library integrated with PyTorch, and I'm encountering a significant challenge in achieving optimal GPU memory utilization and serving efficiency under concurrent load.
1- The Specific Challenge: What are the recommended strategies or best practices for implementing resource aware serving? How can one effectively manage and dynamically allocate GPU memory and compute resources between models of different sizes running simultaneously on the same hardware?
2- Technical Implementation Focus: I am looking for insights on how to leverage or integrate advanced techniques like Batching (across models), Paging Attention (vLLM's core mechanism), or Tensor Parallelism efficiently in this specific multi model, multi-size serving setup.
3- Tooling Recommendation: Are there specific Python libraries or frameworks ( vLLM, Text Generation Inference, Triton Inference Server, or Ray) that integrate seamlessly with the Hugging Face/PyTorch ecosystem and are superior for this exact scenario compared to a standard transformers pipeline, especially when balancing high resource efficiency with flexibility in model size and request handling?
Any detailed code examples or references to proven architectures would be greatly appreciated.
