In late October 2025, NHR@FAU launched its experimental “LLMs as a Service“—because we have the technical expertise to do so and recognized the urgent need for such a service. There has not been any funding for these activities and, therefore, there are no formal obligations. Nevertheless, we did our best—and the usage figures speak for themselves.
In the meantime, we issued more than 200 API keys—70% to FAU members. Usage continues to grow rapidly. What an incredible success, given that the service has never been actively promoted!
On a good day, the service receives up to 1,000,000 (1 million) requests and delivers more than 2,000,000,000 (2 billion) tokens. Not all models are equally expensive—but to put this into perspective: OpenAI charges $5 per million input tokens and $15 per million output tokens for their high-end models. The average price for gpt-oss-120b on OpenRouter is currently still $0.10/$0.40. With Anthropic’s $20 team seat, power users hit rate limits very quickly. Thus, two billion tokens in a single day represents considerable value and, hopefully, a meaningful contribution to science. As we do not log the content of requests, we have no insight into how people use the service. Usage is also very unevenly distributed across all issued API keys. The Top 10 users consume far more than the remaining 95% keys.
The service has no institutional founding yet. Therefore, we are more than grateful to our individual sponsors who make this service possible by allowing us to use hardware they purchased from their own funds:
- A group from HS Hof has made hardware they funded in Helma available to the service (4× data center NVIDIA Hopper GPUs).
- A group from the University of Würzburg has made hardware they funded in Helma available to the service (2× data center NVIDIA Hopper GPUs).
FAU’s contribution so far is an entry-level NVIDIA L4 GPU from the FAU’s OpenStack compute cloud.
Over the past weeks and months, many additional GPUs had to be added again and again to the LLM service to expand the available models, to include large models, and to improve response times for busy models. These additional resources have usually been provided temporarily by the director’s budget, which normally serves NHR porting projects.
Nevertheless, the service regularly hits its capacity limits, with some requests taking up to 45 minutes to complete—far from what users would reasonably expect from an LLM inference service.
Consequently, throttling and access limitations have been in effect since June 22. At the same time—as announced in mid-May—many extended models (e.g. mistralai/Mistral-Medium-3.5-128B, moonshotai/Kimi-K2.6, NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4, MiniMaxAI/MiniMax-M3-MXFP8) had to be removed, as no further hardware sponsors have been found so far and the director’s budget cannot be stretched indefinitely to cover the LLM service. We are aware that these cutbacks hit researchers and students from institutions with no or minor investments hard but we have seen strong increase of utilization from the major sponsoring institutions. We appreciate your understanding.
You may wonder: NHR@FAU operates such a large number of GPUs—so why do they claim a shortage of hardware resources? The answer is straightforward: almost all of our systems are funded through grants with specific regulations (NHR, BayernKI, Basismodell, money from UTN and other universities) that do not permit dedicating nodes permanently to an LLM inference service. We only did so over the past months as a proof of concept to allow people to better see the possibilities and limits of local AI. As our HPC users are well aware, all of our systems are already running at full capacity under regular batch operation. Consequently, there is no idle capacity to set aside.
Are high-end data center GPUs required to run larger LLMs?—It depends. Deepseek-V4-Flash fits into two NVIDIA DGX Spark MiniPCs. Good? Guess the number of DGX Spark it would need to deliver the token rate we already see every day …
Keeping an LLM inference service operational—as we have done over the past six months without any institutional support—requires institutional commitment and substantial investments. To date, the entire service has been run as a side project, alongside—and in addition to—all our regular responsibilities. A cost-recovery model that passes expenses on to users does not appear to be the right approach: every application for a new API key would require approval by a cost center owner, students would be excluded, and additional staff would be needed to set up and maintain a billing infrastructure.
Parts of the text have been polished using AI. The teaser image is fully AI generated.
Testimonials—What heavy users of the service say:
I have used the NHR@FAU LLM service extensively for research and development, particularly for building agentic AI systems and experimenting with large language models. The service has enabled me to prototype and develop multi-agent workflows, automate complex tasks, and explore applications involving fine-tuning, retrieval-augmented generation, and AI-assisted software development.
Having access to a reliable institutional LLM service has significantly accelerated my development and research workflows. It has lowered the barrier to experimenting with new ideas and allowed me to focus on building and iterating rapidly without the overhead of managing dedicated infrastructure. I rely on the service regularly, and continued institutional support would be highly beneficial for researchers and students working with modern AI technologies
I am sincerely grateful for providing this powerful service. It has been instrumental in my thesis research, primarily by giving me access to large-scale models such as DeepSeek-V4-Flash that would otherwise be out of reach.
I am currently using the service to develop a RAG pipeline for a system that delivers personalised responses to students. My research focuses on overcoming the ‘one-size-fits-all’ limitation of standard LLMs—a critical challenge in education. Since every student has a unique academic background, a generic response is rarely optimal. This service allows me to explore approaches that ensure each user receives a tailored, accurate, and personalised answer.
The service has been a great experience. I used it primarily to experiment with different models and identify which performed best for my Master’s thesis research. Not having to deploy and manage the LLMs myself was a significant relief during development. Even though I already had access to NHR clusters, the endpoint reduced my workload considerably.
I also used it for personal workflows out of curiosity—to see how local models compare to the large closed-source ones. I found myself asking questions and uploading documents, such as letters, that I would not have sent to a proprietary service for data privacy reasons. This encouraged me to consider local models for a larger part of my work, and gave me a much better understanding of their capabilities for everyday tasks.
The LLM endpoint of NHR@FAU has proven genuinely useful for my research. It directly enabled a paper benchmarking the endpoint’s models against commercial providers—accepted at a conference and due to appear in August (arXiv:2605.12772). Running experiments through the endpoint saved me significant costs I would otherwise have paid in tokens to commercial providers.
That said, my experience with very large models has been mixed. The Kimi model (1T parameters) struggled in interactive use—sessions would stall in a queue, and the model itself proved weaker than expected, refusing larger tasks that other models handled without issue.
This raises an open question for me: where exactly is the sweet spot for a centralised inference service? Models up to 40–50B parameters can be hosted locally with reasonable hardware, and 120B models are increasingly viable on high-end consumer machines. The clear advantage of the endpoint remains data privacy—keeping everything on premise. But it would be worth thinking carefully about which model sizes and use cases truly benefit from a central instance, particularly for long-running interactive sessions.
The Kimi K2.6 model with 1T parameters has been running on 8× RTX Pro 6000 BSE. While the parameters barely fit into the available VRAM, this leaves little room for the KV cache, and the memory bandwidth of the RTX Pro 6000 BSE is considerably lower than that of high-end GPUs such as the H200 or B200. This likely explains at least part of the performance issues described above.
The LLM service at the NHR@FAU center has, for the first time, made it possible for us to process the structural-materials literature at scale rather than paper by paper. Running the inference across tens of thousands of publications, we can now consolidate exactly the kind of knowledge that has was impossible to manually analyse: the synthesis routes, processing sequences, and microstructural-engineering strategies that are described in text rather than captured in numbers. Distilling these qualitative, methods-level insights from the literature into structured, queryable forms was simply not possible before with conventional text mining. Having this capacity available locally, close to our data and without per-token cost, has turned what would have been a multi-year manual effort into information available on demand. It is opening up a completely new way of working with the materials-science corpus at scale, revealing contradictions and blind spots and it greatly helps us in probing new research ideas.
The NHR@FAU LLM service has been central to our research on automated identification of sensitive information in medical data and for doing research on VLMs directly for image anomaly detection. We use the hosted models to detect and flag protected health information across large clinical text and imaging-report corpora at a scale and consistency that manual annotation by human reviewers cannot match. Beyond throughput, the local deployment is what makes this work possible at all and and defines an example how it could be done in a clinic as well. In our evaluations the automated pipeline is not only faster but also safer than human review, because it does not tire, applies the same criteria to every record, and removes the need to expose sensitive material to additional human readers during the de-identification step itself. Having capable models available as on-premise infrastructure has turned medical data de-identification from a manual bottleneck into a reproducible, auditable process.
The LLM service of NHR@FAU has become an invaluable resource for a wide range of research projects at the Faculty of Humanities, Social Sciences, and Theology. In close collaboration with researchers, we have supported the application of the service to tasks such as morphological annotation of experimental stimuli, automatic detection of commercials in TV subtitle data, sentiment analysis of Middle High German texts, and topic clustering of telephone conversations among others.
Beyond its versatility, one of the service’s greatest strengths is that it is operated locally at FAU. This enabled researchers to work with sensitive and copyright-protected data while maintaining high standards of data protection and compliance. The combination of powerful language models, ease of access, and secure local deployment has significantly lowered the barrier for researchers in the humanities and social sciences to integrate LLMs into their research workflows and apply state-of-the-art AI methods in their work.
He used almost 5.000.000.000 (5B) token in the past 4 weeks. That’s more than 15% of all tokens. But finally switched back to the latest commercial agents because they are simply better.
I research autonomous software agents and their capabilities for decision making using LLMs. As different LLMs grant different levels of competency to agents, there are a lot of experiments necessary to reliably research where LLM agents are useful and how they can be developed and deployed in future, complex
systems. Over the last months, the LLMs as a Service at NHR@FAU has given me the opportunity to experiment with state-of-the-art open-weight models with a low entry barrier and gain deep insights. Instead of being limited by private hosting or worrying whether my inputs are used to train commercial models, I can focus on the quality and usefulness of my research. I thank the NHR@FAU for hosting their invaluable LLM service. I cannot imagine doing my work without it.
I am sincerely grateful for the LLM-as-a-Service provided by NHR@FAU, which has supported my research on benchmarking LLM-based web agents for web navigation.
As part of this study, I created a challenge inspired by the Wiki Game, in which agents had to navigate from a starting point to a target point using different types of resources. As the navigation strategy is determined by the underlying language model, evaluating different models across the challenge was a crucial part of the study.
The broad selection of models provided by the LLM-as-a-Service allowed me to conduct systematic experiments and identify patterns that contribute to more effective resource representations.
Throughout the past months, the platform has been an invaluable resource for exploring how different language models affect agent behavior and performance.
The NHR@FAU LLM service has significantly changed what we can do as a German NLP research group. Because the service hosts frontier open-weight models on-site, we can finally generate high-quality German synthetic datasets for language-model training, a resource that is otherwise painfully scarce and one we hope will help move German NLP research forward.
It is also a valuable teaching tool we couldn’t otherwise provide, as students in our practical courses can now get hands-on experience using cutting-edge models for NLP research (such as large-scale classification tasks) and to check their own data and results.
We’re extremely grateful this exists, and we’d love to see it supported for the long run.
For our everyday applied NLP work, the biggest win of the NHR@FAU service is the simple fact that nothing is logged and nothing leaves Bavaria. That makes it the first LLM backend we can confidently point even at more sensitive material, allowing us to run retrieval-augmented search over our own internal chair knowledge base (wikis and notes) with local embedding models, something we would never send to a commercial API.
We’ve also built it into real application pipelines: a related-work bot that automatically reads, pre-screens, and clusters papers by embedding, and our BibSonomy spam-classification pipeline, which uses the fast, smaller models to filter unwanted submissions at scale. It also quietly powers a lot of our day-to-day work, from agentic coding via OpenCode to automatic meeting summaries.
Having capable models available as on-premise university infrastructure, rather than scattered across private commercial accounts, enables so many novel applications and exploratory research.
I’m very grateful that the NHR@FAU offers the “LLM as a service” because this helps us to complement our own local ChatBots with additional capacity and more capable models. Especially for agentic programming tasks, the resources provided by NHR@FAU are invaluable for us, since very few users are sufficient to completely utilize our own resources in those agentic scenarios. For this kind of use case, the API-based access is the normal way, so missing chat UIs are no problem.
I’ve used the DeepSeek v4 Flash model a lot, because it provided the best compromise between speed and result quality. I also tried Kimi K2.6, but it was not that much better that it rectifies the much slower speed. I’m looking forward to test Minimax M3 this week and hope for an exchange of Kimi K2.6 against GLM 5.2, the currently best open weight model.
We will have to see what can be done. GLM 5.2 requires significantly more resources—16× H100 in FP8.
The LLM-as-a-Service offering provided by NHR@FAU has become an extremely valuable resource for our work. In particular, the API-based access to powerful foundation models allows us to extend our own AI systems with additional capacity and advanced capabilities that would otherwise require significant local infrastructure investments.
The service has proven especially beneficial for agent-based software engineering and coding workflows. In these scenarios, even a small number of users can generate substantial computational demand. Having access to the resources provided by NHR@FAU enables us to conduct ambitious experiments and development projects that would otherwise be difficult to realize. Since our workflows are designed around API integration, the absence of a dedicated chat interface is not a limitation for our use cases.
At the Faculty of Computer Science, we are actively exploring how AI-assisted and agent-based software development can be incorporated into both research and teaching. Continued support from NHR@FAU in this area would significantly strengthen these efforts and help us further develop innovative educational and research activities.
I have extensively used the provided models to develop and evaluate complex demonstration applications. Using these models, it was possible to create applications exceeding 100,000 lines of code, with remarkably strong results after only moderate refinement and iteration. These experiences have provided valuable insights into the potential of modern large language models for software engineering. Many of these investigations and practical evaluations would have been difficult, if not impossible, without access to the NHR@FAUservice.
The API service delivers substantial value for research, teaching, and innovation, and has become an important enabler for exploring the next generation of AI-assisted software development.
“LLM as a service” is great! I’m using it very often for implementations tasks (research prototypes, demonstrators for students) using OpenCode, but also just as a backend for my own AI applications.
I am currently using one of the large models as a browser agent in my automated literature review workflow — analysing publications, extracting relevant metadata, and supporting the screening phase.
The service is particularly valuable given the large number of publications I need to screen, and the model has performed well for this task. I initially tried gpt-oss-120b, but found MiniMax-M3-MXFP8 to be the stronger performer. With a substantial number of publications still to process, I am heavily dependent on continued access to this service.