HUST'26 · SC26 · Chicago · November 15, 2026
$ module load uv$ srun uvx thinguv-manager
A Transparent Wrapper for Safe, Architecture-Aware Python Environments on Heterogeneous HPC Clusters
Geoffrey Lentner · Purdue UniversityLev Gorenstein · Globus – University of Chicago
This talk presents uv-manager, a transparent wrapper around the uv package manager that makes it safe to deploy on HPC clusters whose nodes differ in CPU architecture and whose scratch storage is purged. The paper is by Geoffrey Lentner of the Rosen Center for Advanced Computing at Purdue University and Lev Gorenstein of Globus at the University of Chicago, and was presented at the HUST'26 workshop at SC26 in Chicago on November 15, 2026.
Common ground
Moving away from Anaconda
CaRCC white paper · June 2026
Addressing the Challenges Posed by Anaconda's New Licensing Model
Alber, Creech, D'Auria, Gorenstein , Joshi, Kaufman, Lentner , Raney, Romero, Warriner
doi:10.5281/zenodo.20694933
Revised terms made the licensing status of Anaconda's default channels something each institution had to establish.
The working group documented, across member institutions:
compliance scope budget exposure workflow disruption
uv is one of the places that migration went.
In 2025 and 2026 Anaconda's revised terms of service made the licensing status of its default channels something academic institutions had to establish for themselves. A working group of the Campus Research Computing Consortium (CaRCC) documented the compliance scope, budget exposure and workflow disruption this caused across its member institutions, in a white paper published in June 2026 (doi:10.5281/zenodo.20694933). Both authors of this talk are among its ten authors. Centers that had standardized on a curated conda distribution acquired a migration they had not planned, and the uv package manager is one of the places that migration went.
Common ground
Many ways to build Python environments on HPC
Built by the center
Built by the user
LIVES FOR YEARS
LIVES FOR ONE JOB
Build and curate Spack · EasyBuild · modules · curated conda
The stack cannot keep pace with the long tail, and curation never ends.
Bring your own containers · personal conda · pip --user
Users learn a second toolchain, or fill their home quota.
Built per job Too slow, until now.
High-performance computing centers build Python environments in many ways, and they sort along two lines: who builds the environment, and how long it lives. The center can build the software stack itself with Spack or EasyBuild and expose it as modules, or deploy a shared conda distribution with curated environments; such stacks live for years, cannot keep pace with the long tail of packages users need, and carry a curation obligation that never ends. Users can bring their own, as a container image, a personal conda installation or pip installs into the home directory; the first means learning a second toolchain, and the others fill the home quota. An environment built for each job, by the user, used to be too slow to consider. The next slide explains why uv changes that.
Common ground
uv makes a per-user environment cheap
One binary no bootstrap interpreter
It ships as one self-contained binary.
Fast built per job
It builds an environment inside a job's own setup.
Locked uv.lock · PEP 751
A lockfile from March resolves the same way in September.
Brings its own Python uv python install
It installs the interpreter a project asks for.
Inline scripts PEP 723 · uv run
A script declares its dependencies and runs with them.
Tools on demand uvx · uv tool install
uvx runs a command-line tool with no install step.
We've been recommending uv to our users, but resisted a shared module.
The uv package manager, from Astral, changes the trade-offs behind those answers. It ships as one self-contained binary, so it needs no Python interpreter to bootstrap it. It resolves and installs fast enough to build an environment during a job's own setup, so environments no longer need to be long-lived, curated and shared. Its universal lockfile pins a resolution so that an environment built in March resolves the same way in September, and it exports to the lock format defined by PEP 751. It also installs and manages Python interpreters itself, runs single-file scripts that declare their dependencies inline under PEP 723, and runs command-line tools on demand through uvx or installs them with uv tool install. Our center had been recommending uv to users for some time, but had held back from offering it as a shared module.
The driving case · Anvil MEP
Submit a function
client
anywhere · Python 3.12.8
Globus Compute
the cloud service
Anvil MEP
multi-user endpoint · Python 3.13.13
from globus_compute_sdk import Executor def analyze(n): import numpy as np return np.random.default_rng(n).standard_normal(10**6).std() with Executor(endpoint_id=ANVIL_MEP, user_endpoint_config={"requirements" : "numpy" }) as ex: print(ex.submit(analyze, 42).result())
1 It runs anywhere with the SDK and a Globus login.
2 The requirements travel with the task.
3 The result returns as a future.
The Anvil multi-user endpoint (MEP) is a public Globus Compute endpoint on Purdue's Anvil cluster. A client anywhere, with the Globus Compute SDK and a Globus login, submits an ordinary Python function through an Executor that names the endpoint; ANVIL_MEP stands for the endpoint's identifier. The function's package requirements travel with the task in the user endpoint configuration, here a single requirement, numpy. The call returns a future, and the result arrives when a worker on the cluster has run the function. The next slide shows what the endpoint does with that requirements string.
The driving case · Anvil MEP
Automatic environment provisioning
client
anywhere · Python 3.12.8
Globus Compute
the cloud service
Anvil MEP
multi-user endpoint · Python 3.13.13
{% set major, minor, micro = user_runtime.python.version_tuple %} ENV_HASH=$(echo {{ requirements }} | md5sum | awk '{print $1}') ENV_PATH="$ENV_SITE/py{{ major }}.{{ minor }}.{{ micro }}-$ENV_HASH" uv venv "$ENV_PATH" --python {{ major }}.{{ minor }}.{{ micro }} source "$ENV_PATH/bin/activate" uv pip install globus-compute-endpoint=={{ sdk_version }} uv pip install -r <(echo -e {{ requirements }} )
1 The client's Python and SDK versions arrive with the task.
2 The same requirements string, hashed, names the environment.
3 uv builds it on first use; later calls reuse it.
The Anvil multi-user endpoint (MEP) is a public Globus Compute endpoint on Purdue's Anvil cluster. A client anywhere submits a Python function through the Globus cloud. The endpoint starts a user endpoint on the user's behalf, matched to the client's Python version, and its workers run the function on the cluster. The endpoint's worker_init template reads the client's Python version and Globus Compute SDK version, hashes the function's requirements to name a virtual environment, and builds that environment with uv just in time, so the user never logs in to prepare anything. The requirements string highlighted in the template is the one the client sent on the previous slide. The versions shown are those of the September 2026 demonstration.
The driving case · Anvil MEP
The good, the bad, and the ugly
Our own slide, September 2026
It installed uv to ~/.local/bin.
It did not guard against user configuration.
It filled /home without bound.
It met x86_64 login nodes and aarch64 compute.
Three assumptions a cluster breaks
A1 Home has room for the cache. Ours has a 25 GB quota.
A2 A person with a shell profile is present.
A3 Every node shares one CPU architecture.
Exec format error
The architecture case fails deep in the job log, after the allocation is charged.
The template on the previous slide worked, and it also showed every way a cluster breaks uv. It installed uv into the home directory, it did nothing to guard against a user's own uv configuration, it let caches and environments fill the home quota without bound, and it ignored that Anvil's login nodes and some compute nodes differ in CPU architecture. These map onto three assumptions uv makes that a cluster violates: that the home directory has room for its state, that an interactive user with a shell profile is present, and that every node shares one CPU architecture. The slide orders them from the most familiar to the least; the paper numbers the architecture assumption A2. The architecture case is the expensive one, because it fails with Exec format error deep in a job log, after the allocation has been charged.
Other centers
Home vs scratch storage
Home Scratch Center Quota (GB) Inodes Purge (days) Keyed on More than one ISA Anvil and RCAC community clusters 25 1–2 M 30–60 atime, mtime yes NERSC 40 10 M 56+ atime · TACC Frontera 25 none 10 atime no CSCS Alps 50 1 M 14, 30 atime yes LUMI 20 2 M · · no Sherlock 15 20 M 90 last data write ·
From each center's published user documentation, August 2026. A dot marks a value the center does not publish. Full table and citations in the paper.
The same constraints appear across the community. The table samples home quotas, scratch purge policies and scratch inode allowances from each center's published user documentation, accessed in August 2026, as reported in the paper's Table II. A dot marks a value the center does not publish. Several centers, ours among them, also run nodes of more than one CPU architecture, which is the mismatch uv-manager exists to handle.
What uv-manager is
A thin layer over the real uv
uv · uvx
the user types the real name
argv[0]
one script, dispatched by the name it was called as
$UVM_ROOT
state root on capacity storage: the module's value, or the first writable scratch; never /tmp
uname -m
platform key, resolved on the node that runs the command
provision
uv for this architecture, on first use, under a lock
exec uv
export the UV_* storage variables and hand over
One shell script stands in for uv and uvx , sets up the environment for this node, and hands over to the real binary.
you still just type uv
Users keep typing uv and uvx. Each call passes through one shell script that decides what it was called as, finds a state root on the cluster's capacity storage, appends the CPU architecture of the node actually running the command, installs uv for that architecture the first time it is needed, sets uv's storage variables, and replaces itself with the real uv. The next slide shows where that leaves each user's files.
What uv-manager is
Why put environments on scratch?
Home · small · never purged
$UVM_DATA/
├── bin/ trampolines
└── tools/<name>/ records Scratch · large · purged · one tree per architecture
$UVM_ROOT/<arch>/
├── versions/<ver>/ uv, uvx
├── cache/ UV_CACHE_DIR
├── tools/ UV_TOOL_DIR
├── python/ UV_PYTHON_INSTALL_DIR
└── bin/shims/ UV_TOOL_BIN_DIR
Treat the environment as a cache that a purge may evict, and keep its recipe on home.
The usual instinct is to put Python environments on persistent storage, and on most clusters that means the home directory, whose quota here is 25 GB. uv-manager splits a user's state in two. Under the home directory it keeps only what cannot be rebuilt: a record of each installed tool and a small trampoline that starts it. Measured on Anvil on November 12, 2026, two installed tools took 1,675 bytes there, 573 bytes per trampoline. Everything that can be rebuilt, which is uv itself, its cache, the tool environments and the managed Python interpreters, goes to scratch, under one directory per CPU architecture: 200 MB in 3,726 files for uv and the same two tools, and a populated tree reaches 1.2 GB and 17,798 files. The environment becomes a cache that a scratch purge may evict, and the records on home are what the repair rebuilds it from. A site or a user with project storage that is not purged can point UVM_ROOT there instead, which the uv-manager README lists as the first mitigation; scratch with repair is for those who want or only have scratch.
What uv-manager is
Resolve the architecture on the node
Resolved at module load
login node
x86_64 · module load
sbatch --export=ALL
compute node
aarch64 · gets x86_64 tools
Exec format error
UV_TOOL_DIR=$SCRATCH/.uv/x86_64/toolsThe modulefile fixed the architecture on the login node.
Resolved on the node
login node
x86_64 · module load uv
sbatch --export=ALL
compute node
aarch64 · appends uname -m
the job runs
with aarch64 tools
UVM_ROOT=$SCRATCH/.uvuv-manager exports a base with no architecture, and the node completes it.
A modulefile is evaluated where module load runs, normally an x86_64 login node, and Slurm's default --export=ALL then copies that environment verbatim onto compute nodes that may be aarch64. Anything the modulefile resolved there is wrong on the compute node: exporting a uv tool directory with x86_64 in its path hands every aarch64 job x86_64 tool environments, which fail with Exec format error. uv-manager exports only an architecture-neutral base, UVM_ROOT, and the wrapper appends the architecture on the node that runs the command, so the job finds the environments built for that node. The wrapper's own override, UVM_PLATFORM, recreates the bug if a modulefile sets it, so it belongs in the job script.
The other half
Can we repair environments on scratch?
Scratch purges each file on its own age
__init__.py
core.py
submodule.py
tests/
RECORD
purge line
time since last read
What uv reports on that environment
uv tool listinstalled uv tool upgradenothing to upgrade uv tool installalready installed mytoolImportError
uv trusts a .dist-info directory and never checks the files beside it. So does pip.
Our first detector read each package's RECORD, which the purge takes first. It reported OK on a stripped tqdm; with RECORD restored, missing 197 of 201 files .
Moving state off the home directory onto scratch creates a second problem, because scratch is purged file by file. A purge engine judges each file on its own age, often its access time, so importing a package keeps the files it reads and lets the rest age out. A used environment therefore loses its rarely read files first, including the package metadata. The sketch shows that pattern and plots no data. Once uv considers a package installed, which it decides from the presence of a .dist-info directory, it does not check the package's files: it reports the tool installed and up to date, runs it, and the user sees an ImportError pointing at their own code. This follows the packaging standards' installed-state model, and pip behaves the same way. Our first detector read each package's RECORD manifest, which is among the first files a purge removes, so on one stripped environment it reported nothing wrong; with the manifest restored it reported 197 of 201 files missing.
The other half
What does a purge leave behind?
One environment after a purge
site-packages/
├── tqdm/
│ ├── __init__.py
│ ├── std.py
│ └── contrib/
│ └── logging.py
└── tqdm-4.66.0.dist-info/
├── METADATA
└── RECORD The directories stay: name and version survive.
The repair, in order
1 Interpreters import json, os, ssl
A damaged interpreter reinstalls at the patch version it had.
2 Installed tools $UVM_DATA/tools/<name>/ · home
Each package reinstalls at the version its dist-info names. A tool gone whole comes back from its record.
3 uv's cache uv run · uvx
Only the damaged packages reinstall, and without the cache, so a damaged wheel cannot leak back in.
Checking a damaged tree took 2.5 to 4.4 s on Lustre.
A purge removes files and leaves directories. After a purge has taken a package's RECORD manifest and its METADATA, its dist-info directory is still there, and its name, such as tqdm-4.66.0.dist-info, still states the package and the version that was installed. uv-manager 0.8.0 repairs from that evidence and from a record it keeps for each installed tool under the home directory, which is never purged. The repair runs in a fixed order, because every tool runs on an interpreter. First, a managed interpreter that is missing a binary, or that cannot import json, os and ssl, is reinstalled at the patch version it already had. Second, a damaged tool environment has every package reinstalled at the version its dist-info directory names, so its versions and its uv receipt do not change; a tool that lost its whole environment is reinstalled from its record, with every dependency pinned to the version it had. Third, in uv's cache, the environments that uv run and uvx use have only their damaged packages reinstalled, without the cache, so a damaged wheel cannot return; an environment the repair cannot vouch for is unlinked for uv to rebuild. The tree shown is illustrative. On the 0.8.0 release-candidate runs on Lustre, checking a damaged tree took 2.5 to 4.4 seconds.
The other half
Where should the repair run?
Where a repair step could run Node As a line before sbatch the login node the user PrologSlurmctldthe controller Slurm SrunPrologthe submitting host the user Prologthe compute node root TaskPrologthe compute node the user, where the site configures it UVM_REPAIR on the dispatch paththe compute node the user
Inside a job the first call checks the tree; every later call in that job costs one stat.
A repair step has to run on the node that will run the work, as the user who owns the files. Each conventional place for a bring-up step misses one of the two. A command before sbatch runs on the login node and repairs the wrong architecture's tree. Slurm's prolog hooks run on the submitting host, on the controller, or as root on the compute node; its per-task prolog runs in the right place as the right user, but only where a site has configured it. uv-manager puts repair on its own dispatch path behind an opt-in variable, UVM_REPAIR, so it runs wherever the command runs. Inside a Slurm job the first call checks the tree and every later call in that job costs one file-existence test.
The other half
One rank repairs, every rank recovers
#SBATCH --nodes=2 --ntasks-per-node=64 export UVM_REPAIR=1 srun uvx thing node 1 · 64 ranks node 2 · 64 ranks
128 ranks on 2 nodes, Lustre scratch
1 repair, by the rank holding the lock
3 s lock held
4 s longest wait
We killed the repairing rank partway through. All 127 waiting ranks timed out and named the lock and its holder. uvm clean lock removed it, and the rerun repaired once and passed on all 128 ranks.
With UVM_REPAIR set, the first call in a Slurm job checks the tree for damage that a scratch purge left behind and repairs it, while every other call in the same job waits on the lock and then runs. In the run shown, 128 ranks on two nodes started the same command on a Lustre scratch filesystem with a damaged tree. Exactly one rank repaired it, holding the lock for 3 seconds, and no rank waited longer than 4 seconds. In a second run the repairing rank was killed partway through. All 127 waiting ranks timed out and reported the lock and its holder, so the failure was visible and named. One command, uvm clean lock, removed the dead lock, and the rerun repaired the tree once and succeeded on all 128 ranks. These are the 0.8.0 release-candidate runs of November 2, 2026.
Evidence
Benchmarks
3.23 ms The wrapper's own work before it hands over to uv.
+17.75 ms A wrapped uv call took 17.75 ms longer than starting an empty shell, which took 269.5 ms on the same node.
8 ms Starting an installed tool through its trampoline. The tool alone starts in 4 ms.
0.6 ms Added to every call while UVM_REPAIR is off. Once a job has checked, each call adds one stat .
17,798 Files in one user's tree, 1.2 GB, with an interpreter, two tools and a warm cache.
1 of 64 Of 64 processes started together on an empty tree, one installed uv, and all 64 ran the same version.
Dispatch, footprint and cold start: release 0.5.0, one x86_64 node, GPFS scratch. Trampoline and repair: release 0.8.0.
These are the wrapper's measured costs. The dispatch figures come from the paper's evaluation, release 0.5.0 on one x86_64 node of Anvil against its GPFS scratch: the wrapper's own dispatch costs 3.23 milliseconds, and a wrapped call costs 17.75 milliseconds more than the shell startup floor of that node, which was 269.5 milliseconds. The trampoline and UVM_REPAIR figures are measured for release 0.8.0 and reported in its README. A populated per-user tree with one interpreter, two tools and a warm cache holds 17,798 files and 1.2 GB, which matters against scratch inode allowances. In a cold start of 64 simultaneous processes, exactly one provisioned uv and all 64 agreed on the version.
Lessons
What bit us: falling down the rabbit hole
Rebuilds drop pins uv tool install --reinstall moved a ==4.66.0 pin to 4.70.0. Only uv tool upgrade --all --reinstall --no-cache kept it.
Locks cannot judge death Our lock tried to decide whether a silent holder had died, and grew to a quarter of the script. Since 0.7.0 a waiter times out, names the holder, and leaves it to a person to clear the lock.
Status first uvm status prints the node's architecture, the uv version, and every directory uv will use: state root, cache, tools, interpreters, records. Staff run it first, and it saves a round trip on every ticket.
In fairness, uv's docs warned us.
Three lessons from building uv-manager. First, uv's rebuild commands do not all repair: a forced reinstall rebuilds the tool but, as uv's documentation says, replaces its version constraints, so a tool pinned at version 4.66.0 came back as 4.70.0; only uv tool upgrade --all --reinstall --no-cache kept the pin, and when a tool's pyvenv.cfg is missing that same command installs into the base interpreter. Second, we built a lock that tried to decide whether a silent holder had died; it grew to a quarter of the script, and release 0.7.0 stopped asking, because a shared filesystem cannot answer that question reliably. A waiter now times out and names the holder, and a person removes a dead lock. Third, the subcommand people actually use is status, which prints every resolved path and where it came from; we would write it first next time.
Back to the MEP
Upgrading the Anvil MEP to uv-manager
worker_init today
export PATH="$HOME/.local/bin:$PATH" if ! command -v uv &> /dev/null; then curl -LsSf https://astral.sh/uv/install.sh | sh fi
worker_init on uv-manager planned
export UVM_ROOT="$SCRATCH/.uv" export PATH="/apps/external/uv/main/bin:$PATH" export UVM_REPAIR=1
Every submitted function gets an environment that is placed, matched to the node, and repaired in the job.
The loop closes back at the Anvil multi-user endpoint. Its current worker_init template installs uv itself into the home directory. The planned change points the template at the site's uv-manager installation with a state root on scratch and repair turned on, so every function a client submits gets an environment that is placed, matched to the node's architecture, and repaired inside the job. The pattern shown follows the Globus Compute section of the uv-manager README; the switch on Anvil is planned and not yet live.
The agentic era
Supercharging MCPs with uv-manager
Globus-MCP · shipped
compute_submit( endpoint_id="<uuid>" , function_source="def analyze(n): ..." , requirements="numpy" , ) Cluster MCP · proposed
python_submit(code, requirements)
a Cluster MCP tool: any code, under the site's Slurm
your agent harness
the user's choice
tool server
Globus-MCP, Cluster MCP, with the site's context
site scheduler
Slurm, or a Globus Compute endpoint in front of it
uv-manager
placed, per architecture, repaired in the job
At one site, users running coding agents were 19.5% of users and submitted 55.8% of jobs (Harvard FASRC, July 2026; preprint).
Coding agents are already heavy users of some clusters: in a July 2026 measurement at Harvard's FASRC cluster, users running coding agents were 19.5% of users but submitted 55.8% of jobs (Zheng et al., arXiv:2609.38723, a preprint). An agent that has to manage Python environments itself spends its context on caches, quotas and architectures. A site-configured wrapper takes that work off the agent: it can be told that uvx is safe on this cluster. Globus-MCP already exposes compute_submit, which sends a function and its requirements to a Globus Compute endpoint, as on the earlier slides. The sketched python_submit is a proposed tool for Cluster MCP, a separate tool server that works through the cluster's own scheduler: it would take arbitrary code and run it under Slurm, with uv-manager doing the environment work. The agent harness stays the user's choice; the site provides the tools and the context.
In production
The default uv on five Purdue clusters
$ module load uv$ uvm statusuv-manager: 0.8.0 architecture: x86_64 state root: /anvil/scratch/glentner/.uv (from UVM_ROOT) resolved version: uv 0.13.6 (x86_64-unknown-linux-gnu) UV_CACHE_DIR: /anvil/scratch/glentner/.uv/x86_64/cache UV_TOOL_DIR: /anvil/scratch/glentner/.uv/x86_64/tools UV_PYTHON_INSTALL_DIR: /anvil/scratch/glentner/.uv/x86_64/python trampoline dirs: /home/glentner/.local/share/uvm/bin /anvil/scratch/glentner/.uv/bin uv-manager 0.8.0 is the default uv module on Anvil, Gautschi, Gilbreth, Negishi and Scholar, with UVM_REPAIR set for batch jobs.
docs.rcac.purdue.edu
uv-manager is in production at Purdue. Release 0.8.0 is the default version of the uv module on five RCAC clusters, Anvil, Gautschi, Gilbreth, Negishi and Scholar, and release 0.7.0 remains available. The module sets the state root to $SCRATCH/.uv and sets UVM_REPAIR=1, so the first uv or uvx call in a Slurm job checks for purged files and repairs them. The code shows uvm status on an Anvil login node: the wrapper's version, the node's architecture, the state root on scratch, the uv it resolved, and uv's cache, tool and interpreter directories inside the per-architecture tree on scratch. The only directory on the home filesystem holds the tool trampolines. The full output also lists the real uv and uvx binaries, the installed uv versions, the shim directories, the record directory UVM_DATA under the home directory, and any version pin. The first QR code opens the RCAC blog post that explains why the center provides uv through uv-manager, at docs.rcac.purdue.edu/blog/2026/11/01/uv-manager/; the second opens the uv entry in the RCAC application catalog, at docs.rcac.purdue.edu/software/apps_md/uv/, which lists the versions on each cluster. Each cluster's user guide has a page on Python with uv.
uv-manager · HUST'26
Run uvx on any node, after any purge.
uv-manager places the environment, matches it to the node, and repairs it in the job, for a person at a terminal or an agent behind an MCP server.
Geoffrey Lentner · Purdue UniversityLev Gorenstein · Globus – University of Chicago
github.com/PurdueRCAC/uv-manager
MIT license
uv-manager is a transparent wrapper that lets a center offer uv on a heterogeneous cluster. It places each user's uv state on capacity storage, keyed by the CPU architecture of the node that runs the command, and it repairs what scratch purges remove, inside the job that needs it. Both decisions are made on the node that does the work, which is what makes them correct for a person at a terminal and for an agent submitting work through an MCP server. The software is open source under the MIT license at github.com/PurdueRCAC/uv-manager. The paper's co-author is Lev Gorenstein of Globus at the University of Chicago.
