I am currently building an production grade application, where we make multiple llm calls inside the custom llm. We are working on to improve the speed. any advice would be good. There are the options I find valuable
- Using BackgroundTask ( sry, we are using FastAPI )
- Semantic Cache with redis
- Map Reduce ( not sure ~ )
- MCP Servers ( not sure ~ )
Any suggestions would be more helpful. What do you think?