Many organisations in banking, insurance, the public sector and defence see the opportunities large language models (LLMs) offer but can't send their data to external services. The solution is to run the model in the organisation's own data centre. Once difficult and expensive, this option is far more accessible today thanks to high-performing open-weight models and efficient serving software.
1. Start with use cases and volume
Hardware and model choices depend on what you want to do and how much it will be used:
- Which tasks? (document summarisation, question answering, classification, code generation)
- How many users, how many concurrent requests?
- How long are the documents? (context window needs)
- What response time is expected?
GPU investment made without answers to these questions ends up either too small or unnecessarily large.
2. Choosing a model
The open-weight model ecosystem is evolving fast. Consider:
- Task performance: public benchmarks are a guide, but the real test should be on your own data.
- Language performance: if you work with documents in languages such as Turkish, assess the model's ability in that language separately.
- Size and architecture: smaller models are cheaper and faster, and may be enough for many enterprise tasks. Mixture-of-experts (MoE) architectures run at lower compute cost than their total parameter count suggests.
- Licence: commercial-use terms vary from model to model.
For many organisations, the most efficient approach is to combine models of different sizes for different tasks.
3. GPU sizing
The basic rule is simple: memory needed for model weights ≈ number of parameters × bytes per parameter.
| Precision | Per parameter | For a 70-billion-parameter model |
|---|---|---|
| FP16 / BF16 | 2 bytes | ~140 GB |
| FP8 / INT8 | 1 byte | ~70 GB |
| INT4 | 0.5 bytes | ~35 GB |
On top of this, leave roughly 20–40% headroom for activations, the KV cache needed for concurrent requests and serving overhead. Long documents and many concurrent users increase that headroom.
Quantisation reduces memory needs substantially by lowering the model's precision. Quality loss is usually limited, but test it on your own tasks.
4. Serving software
The software used to expose the model as an API directly determines performance. As of 2026, vLLM has become the de facto standard for on-premise LLM serving, with efficient memory management, continuous batching and multi-GPU support. Because it offers an OpenAI-compatible API, moving applications between cloud and on-premise models also becomes easier.
5. Architecture: the model alone isn't enough
A usable on-premise AI platform usually includes:
- A model serving layer (e.g. vLLM)
- RAG infrastructure: document processing, a vector database, retrieval
- Access management: enterprise authentication, role-based permissions, document-level access control
- Safety layers: input and output checks, sensitive data masking
- Logging and monitoring: usage, performance, cost and quality
- Evaluation: measuring quality whenever the model or prompts change
6. Cost: when does on-premise win?
On-premise cost comes from hardware, energy, data centre space and the operations team. API services charge per use.
Industry analyses suggest that at high, sustained usage (for example around 1 billion tokens a month or more, with GPUs busy most of the time), owning infrastructure can beat APIs. At low or irregular usage, APIs are usually more economical.
For many organisations, though, the real decision is driven not by cost but by regulation and data security. If data can't leave the building, the question isn't "which is cheaper?" but "how do we do this most efficiently in-house?"
Deployment steps
- Define use cases and volume
- Test candidate models on your own data
- Size hardware for the use cases and growth plan
- Set up the serving, RAG, security and monitoring layers
- Validate quality with an evaluation set
- Start with pilot users and roll out gradually
Conclusion
On-premise LLM deployment is no longer something only big tech companies can do. With the right model, the right sizing and the right architecture, you can get real value from GenAI without your data leaving your premises.
Through our On-Premise AI Infrastructure service we support you from requirements analysis and hardware sizing to model selection and operations. For technical teams, our Building LLM Applications programme also covers on-premise serving with vLLM.
Sources