How to Plan On-Premise AI Computing for National Healthcare

When a country decides that its citizens' health data must be processed at home, the next question is how much computing capacity that takes. The wrong answer is expensive either way. Too little, and a stroke patient's CT scan waits in a queue behind a research job. Too much, and public money sits idle. Capacity for national healthcare AI is not one number. It is the sum of several workloads with different shapes, plus redundancy, test environments and room to grow. This article sets out a practical method for planning it, using the scope of Moldova's National Healthcare Digitalization Program as the worked example.

Step 1: separate the workload types

Healthcare AI combines three types of computing workload. They behave so differently that they should be planned separately.

Workload	Typical examples	What drives sizing
Real-time inference	Imaging triage, drug-interaction checks at prescribing, deterioration alerts	Peak load and a latency target. Capacity must exist when the request arrives.
Training and fine-tuning	Adapting models to national data, periodic retraining, validation runs	Model size, dataset size and how often retraining is needed. Jobs can be scheduled.
Batch processing	Genome analysis, overnight population risk scoring, retrospective re-reads, forecasting runs	Total volume per day or month. Throughput matters more than latency.

The key distinction is between latency-bound and throughput-bound work. A batch job can run at night or wait an hour. A suspected intracranial haemorrhage cannot. Planning that pools everything into one average figure hides this difference, and clinical service levels are what suffer.

Step 2: work out memory residency, not just compute

For modern AI models, the binding constraint is often accelerator memory rather than raw arithmetic speed. A model can only answer quickly if its weights are already loaded, or resident, in accelerator memory. Loading a large model from storage can take minutes, which is unacceptable in a clinical workflow.

A simple rule of thumb for inference

Model weights need roughly parameters × bytes per parameter. At 16-bit precision that is 2 bytes per parameter, so a 70-billion-parameter model needs about 140 GB for its weights alone. Quantisation reduces this. The LLM.int8() work showed that 8-bit inference can halve memory requirements with no loss of performance on the models it tested, and lower-precision formats go further, with trade-offs that must be validated clinically.

Weights are only part of the picture. Language and multimodal models also need a key-value cache for every active request. The authors of the PagedAttention paper describe this memory as "huge" and note that it grows and shrinks dynamically with each request. More concurrent users and longer inputs, such as a full patient history, mean more cache memory. Efficient serving software helps a great deal: PagedAttention improved throughput by 2–4 times at the same latency in the authors' tests. But the cache still has to be planned for.

Training needs far more

Training is more memory-hungry. The ZeRO paper shows that mixed-precision training with the Adam optimiser needs about 16 bytes per parameter for weights, gradients and optimiser state, before counting activations. Fine-tuning even a mid-sized model therefore needs several times the memory needed to serve it. Techniques such as sharding, parameter-efficient fine-tuning and gradient checkpointing reduce this, but training capacity is still a separate planning item.

Step 3: give parallel clinical workloads dedicated capacity

It is tempting to build one shared pool and let a scheduler hand out resources on demand. For a national healthcare programme with several clinical domains, that approach creates real risks:

• Memory contention. If two large models cannot both stay resident, one gets evicted and must be reloaded when its next request arrives. The resulting delay breaks latency targets.
• Noisy neighbours. A large training or batch job can saturate memory bandwidth, interconnect or storage and slow down inference running alongside it.
• Unclear accountability. When imaging triage misses its target, the operator needs to know whether the cause was the model, the data or a genomics job on the same hardware.
• Change control. Medical AI needs controlled, documented updates. The FDA's guidance on predetermined change control plans, for example, expects manufacturers to describe how model modifications will be developed, validated and implemented. Separate environments per domain make such changes easier to isolate and audit.

The practical result is dedicated capacity per clinical domain, sized for that domain's peak, with real-time inference kept apart from training and batch work within each domain. Shared pools are still useful for development, research and flexible batch work.

Step 4: build in redundancy

Hardware fails, and maintenance must happen. A clinical service with no spare capacity will have outages. The standard approach is N+1: size for peak demand with N units, then add one so that the service survives the loss of any single unit. Google's Site Reliability Engineering guidance goes further for critical services and recommends N+2, so that peak load can be handled while one unit is down for planned maintenance and another fails unexpectedly.

For healthcare, the sensible pattern is:

• N+1 at minimum for every latency-critical inference service, such as imaging triage;
• standby or backup capacity that can take over a failed inference role quickly, with the models already deployed;
• graceful degradation for batch and training, which can pause and resume rather than needing spare capacity of their own.

Step 5: do not forget development, testing and validation

Every model update must be tested before it reaches patients. That needs an environment that mirrors production, where new versions can run in silent mode against real workflows, be compared with the current version and be signed off. Running these tests on production systems puts the service at risk, and leaving them out puts patients at risk. Reserve and dev/test capacity also does a second job: it can be repurposed in an emergency.

Step 6: plan for data growth

National programmes start small and grow. Capacity sized for today's data will be inadequate within a few years if digitisation succeeds. Growth comes from several directions at once:

• more records, as EHR coverage expands;
• richer records, as imaging, laboratory and genomic data are linked to each patient;
• more models, as new clinical uses are approved;
• larger models, as the field moves to multimodal systems.

Plan modular growth: a physical site, power, cooling and networking sized for the end state, with computing capacity added in stages as utilisation is measured.

Worked example: Moldova's programme scope

Moldova's National Healthcare Digitalization Program (2025–2030) covers around 2.36 million people across five AI domains. Applying the method above shows why each needs its own planning line:

1. Medical imaging handles about 350,000–400,000 CT, MRI and PET-CT studies a year, roughly 1,000 a day on average, with critical findings to be triaged within 5–10 minutes. This is the most latency-critical domain. It needs dedicated real-time inference with N+1 protection, plus separate capacity for training, batch re-reads and backup.
2. Genomics aims to sequence around 24,000 genomes, about 1% of the population. This is a throughput workload. It is sized by genomes processed per month and benefits from accelerated analysis pipelines, but it must never compete with imaging for resources.
3. Pharmacology and treatment planning combines a large multimodal model with a separate drug-interaction model. Both must stay resident to answer prescribers in real time, which makes memory residency the dominant sizing factor.
4. Disease prediction mixes scheduled population risk scoring with real-time epidemiological monitoring, and its data volume rises fastest as EHR coverage grows.
5. Healthcare management runs forecasting and anomaly-detection models with modest real-time requirements and regular batch runs.

The growth factor matters most. EHR coverage is planned to rise from under 10% to around 80% by 2030, roughly eight times more data. Capacity, storage and site infrastructure must therefore be planned for the 2030 data volume from the outset, with computing added in stages. All of it must sit inside the country: the programme requires that citizens' medical data is processed only in Moldova, with public cloud excluded. That matches the European direction set by the European Health Data Space, where analysis takes place in secure processing environments rather than through data downloads.

Sovereign capacity in Moldova

SRL "IMPORT MEDICAL" owns and operates sovereign, on-premise, high-performance AI computing capacity inside Moldova for healthcare workloads, and is a technical partner of the National Healthcare Digitalization Program. We plan capacity with the method described here: workloads separated by type, models kept resident, clinical domains isolated, redundancy built in and growth planned from the start. Learn more on our AI infrastructure page, or read about the evidence on AI in radiology and national genomics programmes. To discuss capacity for your institution or programme, contact us.

AI infrastructure on-premise AI capacity planning GPU computing healthcare AI

Sources

1. Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023) — arXiv
2. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers et al., 2022) — arXiv
3. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Rajbhandari et al., 2019) — arXiv
4. Site Reliability Engineering, Appendix: Production Services Best Practices — Google
5. Marketing Submission Recommendations for a Predetermined Change Control Plan for AI-Enabled Device Software Functions — U.S. Food and Drug Administration
6. European Health Data Space Regulation (EHDS) — European Commission, DG SANTE