What does deploying an LLM on your infrastructure mean?

The model weights and inference software run in an environment controlled by your organization. Applications send requests to a private model endpoint instead of a public AI service. Network, identity, storage, and logging policies remain under company control.

This does not automatically make the whole application private. Prompts may still pass through gateways, search systems, logs, analytics tools, or external integrations. The deployment boundary has to include every component that receives sensitive input or model output.

The deployed system

01
Business application

Chat interface, document workflow, RAG system, AI agent, API, or background process used by people and other software.
02
Application services

Authentication, prompt construction, retrieval, tool execution, output validation, and business rules.
03
Private model endpoint

A stable API that accepts requests, manages queues, streams responses, and applies request limits.
04
Inference engine

Software such as vLLM, SGLang, llama.cpp, or TensorRT-LLM that loads and runs the selected model.
05
Models and adapters

Approved model weights, tokenizers, configuration, optional quantized versions, and task-specific adapters.
06
Infrastructure

GPU or CPU compute, memory, storage, containers, networking, secrets, monitoring, and backup configuration.

Where can the system run?

The right location depends on data rules, available hardware, internal operating capacity, and how much demand changes over time.

On-premise

Location: Company data centre, server room, or workstation
Usually suitable for: Strict network isolation, existing GPU capacity, fixed workloads, or sites with limited external connectivity
Main tradeoff: The organization owns capacity planning, hardware availability, power, cooling, and replacement

Private cloud

Location: Your cloud account, VPC, private subnet, or dedicated tenant
Usually suitable for: Private networking with faster provisioning, managed storage, monitoring, and adjustable compute capacity
Main tradeoff: Cloud infrastructure remains an external dependency and idle GPU resources can be expensive

Hybrid

Location: Sensitive components run privately while approved tasks use another environment
Usually suitable for: Workloads with different data classes, model requirements, or temporary demand peaks
Main tradeoff: Routing rules and data classification must be explicit and testable

Do you need a self-hosted LLM?

Private deployment gives more control, but it also creates an operating responsibility. A hosted model API can be the better engineering choice when its data terms and controls meet the requirement.

Deployment on your infrastructure is worth considering when

Prompts, retrieved documents, or outputs cannot leave a controlled network
The application must continue working without an internet connection
Model versions and updates need to follow an internal approval process
Existing hardware can support a stable and predictable workload
The application needs a model or adapter that is not available through the approved hosted services
Request logs, retention, and deletion must remain under direct operational control

A hosted API may be simpler when

The provider's data handling terms already satisfy the requirement
Demand is small, irregular, or changes sharply
The task depends on a model that cannot be deployed under an acceptable licence
The team does not want to operate GPU infrastructure and model-serving software
Fast access to newly released model capabilities matters more than infrastructure control

How the model is selected

The largest model that fits on the available hardware is rarely the correct default. Selection starts with task examples and measurable requirements, then compares models under the intended serving configuration.

Open-weight does not always mean unrestricted use. The model licence, acceptable-use terms, redistribution rules, and any limits on derived models need to match the intended product and organization.

Task quality: Accuracy is measured on the actual languages, documents, output formats, and failure cases of the application.
Model behaviour: Instruction following, tool use, structured output, context length, and multimodal input are checked only when the task needs them.
Licence: Commercial use, distribution, modification, usage restrictions, and required notices are reviewed before deployment.
Hardware fit: Model weights, runtime overhead, KV cache, context length, concurrent requests, and quantization all affect memory use.
Operational fit: Startup time, supported inference engines, update path, monitoring, and the team's ability to maintain the model are part of the choice.

Hardware sizing and capacity planning

A parameter count alone cannot determine the required hardware. Memory use and response speed depend on the model format, precision, input length, output length, concurrency, inference engine, and GPU architecture. The reliable method is to benchmark candidate configurations with a representative workload.

Memory

Enough capacity is needed for model weights, runtime workspace, and the KV cache used by active requests.

Latency

Time to first token and generation speed are measured separately because users notice both.

Throughput

Concurrent users, queued requests, prompt length, and generated tokens determine the sustained load.

Availability

Redundancy, maintenance windows, restart time, and acceptable recovery time determine whether one server is sufficient.

Growth

Expected users, document volume, agent activity, and new applications affect capacity beyond the initial release.

A sizing test should report

• Quality results for each candidate model and quantization
• GPU and system memory use at idle and under load
• Time to first token and output tokens per second
• Throughput and queue time at expected concurrency
• Failure rate, timeout behaviour, and recovery time
• Estimated infrastructure cost at normal and peak demand

Model serving and application integration

A usable deployment needs a stable service around the model. The serving layer controls how applications connect, how requests share the hardware, and what happens when the model or a node fails.

Inference API

A documented endpoint supports streaming, structured outputs, tool calls, request limits, and the response format required by the application.

Quantization

Lower-precision weights can reduce memory use and change speed, hardware compatibility, or output quality. Each candidate is tested on the target task.

Batching and caching

Continuous batching, prefix caching, and KV-cache settings can improve throughput, but their value depends on request shape and traffic.

Parallelism

A model can be split across GPUs or replicated to serve more requests. Multi-node designs also depend on network bandwidth and latency.

Routing

A gateway can direct tasks to different local models, route around an unavailable replica, or send approved requests to an external model.

Application connections

Private RAG, AI agents, internal APIs, desktop applications, and existing products connect through authenticated interfaces.

Security and data handling

Keeping the model endpoint private removes one external data path. It does not remove application vulnerabilities, prompt injection, excessive permissions, unsafe output handling, or compromised model and container dependencies.

Security controls should follow the complete request path, including source documents, vector stores, prompts, model outputs, tools, logs, backups, and administration interfaces.

Network boundary: Private subnets, firewall rules, ingress controls, service-to-service policy, and optional offline operation define which systems can reach the endpoint.
Identity and authorization: Users and services authenticate before inference. Application permissions still apply when RAG or agents retrieve data and use tools.
Encryption and secrets: Connections use transport encryption. Tokens, certificates, registry credentials, and storage keys stay in an approved secret-management system.
Logs and retention: Prompt and output logging is disabled, redacted, or retained according to the data policy. Operational metrics can be collected without copying full content.
Model supply chain: Model files, containers, packages, and revisions come from approved sources, are pinned, scanned, and recorded before release.
Output controls: Applications validate structured output and do not execute model-generated code, queries, or commands without separate checks and permissions.

Who operates the system after deployment?

The operating model is agreed before production. TensorBundle can prepare the system for an internal platform team, provide an agreed maintenance scope, or divide responsibility by component.

Area	Initial deployment	Ongoing work
Models	Evaluate, approve, package, and record the selected revision	Review updates, rerun evaluations, and control rollout or rollback
Serving	Configure inference engine, API, limits, scaling, and health checks	Monitor capacity, latency, errors, queues, and service availability
Infrastructure	Prepare compute, storage, containers, network, and secrets	Apply patches, manage capacity, replace hardware, and test recovery
Application	Connect users, RAG, agents, or products to the private endpoint	Maintain integrations, permissions, test cases, and business behaviour

What you receive

The delivery is a reproducible deployment for the agreed environment and workload. It includes the information needed to run, test, update, and recover the system.

01 Model comparison and hardware benchmark results
02 Documented model and licence selection
03 Container images or installation packages with pinned versions
04 Inference endpoint, authentication, request limits, and health checks
05 Network, storage, secrets, and logging configuration
06 Application, RAG, or agent integrations included in the scope
07 Load tests, quality evaluations, and acceptance results
08 Monitoring dashboards, alerts, backup, rollback, and recovery procedures
09 Deployment and operating documentation for the responsible team

LLM deployment questions

Does the LLM have to run on servers in our office?

No. Your infrastructure can mean on-premise hardware, a private data centre, your own cloud account or VPC, or a dedicated environment controlled under your policies. The required boundary depends on the data and network rules.

Can the system run without internet access?

Yes, if the model files, containers, packages, authentication dependencies, and required data sources are available inside the environment. Updates then need a controlled import process. External tools and hosted models will not be available while disconnected.

Can we use the infrastructure we already have?

Often yes. Existing servers are checked for model compatibility, available memory, storage, network capacity, and current workload. The result shows what can run now and whether an upgrade or different model is needed.

How do we know whether a private model is good enough?

Candidate models are tested on examples from the intended application. The comparison covers answer quality, language, required formats, speed, and important failure cases. A general benchmark cannot replace this task-specific test.

How much does a private LLM deployment cost?

Cost depends on whether hardware already exists, the selected model, expected usage, availability requirements, and who will operate the system. The estimate should separate initial implementation, infrastructure, and ongoing operation.

Do we need an internal AI or infrastructure team?

Not necessarily, but someone must own access, incidents, updates, and capacity after launch. Responsibility can stay with an internal platform team, be included in an agreed maintenance service, or be divided between teams.

Can the model be changed later?

Yes, if the application uses a stable private endpoint rather than depending on one model's details. A replacement model still needs licence review, task evaluation, performance testing, and a controlled rollout.

Will our prompts or documents be used to train the model?

A self-hosted inference service does not train on requests unless a separate training or data collection process is built. Logs, feedback stores, analytics, and backup systems still need explicit retention and access rules.

Can several applications share the same deployment?

Yes. A gateway can authenticate each application, apply separate limits, record usage, and route requests to the appropriate model. Capacity planning must account for their combined traffic and different prompt sizes.

Can we start with a small deployment and expand later?

Yes. A limited first release can serve one model and one application. Expansion may add more users, applications, model replicas, or hardware after actual usage and performance are known.

Can the deployment use both private and external models?

Yes. A hybrid router can keep sensitive tasks on private models and send explicitly approved requests to external services. Routing must use enforceable data rules rather than asking users to remember which information may leave the environment.

What support is needed after launch?

The system needs monitoring, security and dependency updates, capacity checks, model evaluation, backup tests, and an incident process. The exact ownership and maintenance scope are agreed before production.

How long does an LLM deployment take?

A single model on prepared hardware is shorter than a production service with procurement, private networking, high availability, application integration, and security review. A useful estimate requires the workload, candidate environment, data boundary, availability target, and acceptance tests.

Technical references

The deployment guidance on this page follows current primary documentation for model serving, scaling, quantization, monitoring, and LLM application security.

Discuss an LLM deployment