# MinIO AIStor RELEASE.2026-09-07T08-39-31Z Released: 2026-09-07 Tier configuration now recovers on demand instead of staying unloaded for the life of a process, AIStor Tables no longer deletes table metadata when a pointer write reports an ambiguous error, and the RDMA data path gains a zero-copy erasure decode worth roughly 17% on inter-node GET throughput. Inter-node RDMA works on more fabrics, with InfiniBand and IPv6 support and fixes for two conditions that stalled writes under uneven load. The release also carries hardening work from internal review across the IAM, STS and admin paths. --- ## Downloads ### Binary Downloads | Platform | Architecture | Download | | -------- | ------------ | --------------------------------------------------------------------------- | | Linux | amd64 | [minio](https://dl.min.io/aistor/minio/release/linux-amd64/minio) | | Linux | arm64 | [minio](https://dl.min.io/aistor/minio/release/linux-arm64/minio) | | macOS | arm64 | [minio](https://dl.min.io/aistor/minio/release/darwin-arm64/minio) | | macOS | amd64 | [minio](https://dl.min.io/aistor/minio/release/darwin-amd64/minio) | | Windows | amd64 | [minio.exe](https://dl.min.io/aistor/minio/release/windows-amd64/minio.exe) | ### RDMA Binaries Both RDMA features require the `minio.rdma` build. It adds no command-line flags of its own and serves over TCP when no RDMA fabric is present. - **S3 over RDMA (GPU-Direct) is on by default.** Install the `minio.rdma` binary with its bundled libraries on RDMA-capable hardware and it serves GPU-Direct transfers without further configuration. Clients that do not present an RDMA token continue over TCP. - **Inter-node RDMA is off by default.** Turn it on per node with `MINIO_RDMA_INTERNODE=on`. It requires a lossless RoCEv2 fabric with substantial switch-side configuration. This release fixes two conditions that made inter-node RDMA unusable on some fabrics — a window floor that deadlocked a peer, and an even credit split that stalled unevenly loaded nodes — and adds InfiniBand and IPv6 support. See Performance Improvements and Bug Fixes below. | Platform | Architecture | Download | | -------- | ------------ | --------------------------------------------------------------------------- | | Linux | amd64 | [minio.rdma](https://dl.min.io/aistor/minio/release/linux-amd64/minio.rdma) | | Linux | arm64 | [minio.rdma](https://dl.min.io/aistor/minio/release/linux-arm64/minio.rdma) | ### FIPS Binaries | Platform | Architecture | Download | | -------- | ------------ | --------------------------------------------------------------------------- | | Linux | amd64 | [minio.fips](https://dl.min.io/aistor/minio/release/linux-amd64/minio.fips) | ### Package Downloads | Format | Architecture | Download | | --------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | DEB | amd64 | [minio\_20260907083931.0.0_amd64.deb](https://dl.min.io/aistor/minio/release/linux-amd64/minio_20260907083931.0.0_amd64.deb) | | DEB | arm64 | [minio\_20260907083931.0.0_arm64.deb](https://dl.min.io/aistor/minio/release/linux-arm64/minio_20260907083931.0.0_arm64.deb) | | RPM | amd64 | [minio-20260907083931.0.0-1.x86_64.rpm](https://dl.min.io/aistor/minio/release/linux-amd64/minio-20260907083931.0.0-1.x86_64.rpm) | | RPM | arm64 | [minio-20260907083931.0.0-1.aarch64.rpm](https://dl.min.io/aistor/minio/release/linux-arm64/minio-20260907083931.0.0-1.aarch64.rpm) | | DEB(RDMA) | amd64 | [minio.rdma\_20260907083931.0.0_amd64.deb](https://dl.min.io/aistor/minio/release/linux-amd64/minio.rdma_20260907083931.0.0_amd64.deb) | | DEB(RDMA) | arm64 | [minio.rdma\_20260907083931.0.0_arm64.deb](https://dl.min.io/aistor/minio/release/linux-arm64/minio.rdma_20260907083931.0.0_arm64.deb) | | RPM(RDMA) | amd64 | [minio.rdma-20260907083931.0.0-1.x86_64.rpm](https://dl.min.io/aistor/minio/release/linux-amd64/minio.rdma-20260907083931.0.0-1.x86_64.rpm) | | RPM(RDMA) | arm64 | [minio.rdma-20260907083931.0.0-1.aarch64.rpm](https://dl.min.io/aistor/minio/release/linux-arm64/minio.rdma-20260907083931.0.0-1.aarch64.rpm) | Packages are also published under `aistor-*` / `aistor_*` names with identical contents, and as Alpine `.apk`. The installed binary path is unchanged at `/usr/local/bin/minio`. ### Container Images ```bash # Standard docker pull quay.io/minio/aistor/minio:RELEASE.2026-09-07T08-39-31Z podman pull quay.io/minio/aistor/minio:RELEASE.2026-09-07T08-39-31Z # FIPS docker pull quay.io/minio/aistor/minio:RELEASE.2026-09-07T08-39-31Z.fips podman pull quay.io/minio/aistor/minio:RELEASE.2026-09-07T08-39-31Z.fips # GPU-Direct / inter-node RDMA docker pull quay.io/minio/aistor/minio:RELEASE.2026-09-07T08-39-31Z.rdma podman pull quay.io/minio/aistor/minio:RELEASE.2026-09-07T08-39-31Z.rdma ``` ### Homebrew (macOS) ```bash brew install minio/aistor/minio ``` --- ## Security Hardening A number of authorization, credential-handling and resource-bound paths were hardened. Identifiers will be published separately where applicable. ### Authorization and identity - Policy evaluation now fails closed when a policy in a collection cannot be read, merged-policy cache entries are keyed to the source they were derived from, and a session token carries the caller's own policy rather than the requested mode alone (#6776). - A temporary session token is accepted only when presented with the access key it was issued to, on both the S3 SigV4 derived-credentials path and the admin/console bearer-JWT path (#6719). - Shortening an object's governance retention requires `s3:BypassGovernanceRetention`, matching AWS behavior and the DELETE path (#5143). - Claims from a web-identity token reach the STS audit record only after the token's signature verifies (#6673). - Claims from an unvalidated Kubernetes service-account token no longer influence the credential minted for the request (#6572). - Creating an inventory config with an Iceberg target requires the submitter to hold `CreateWarehouse`, `CreateNamespace`, `CreateTable` and `UpdateTable` on that target (#5822). - `minio` and `.minio.sys` are rejected as a batch job source or destination, at validation and again at start, so a job persisted earlier cannot resume against them. ### Information disclosure - `mc admin trace` records headers only on the `inspect-data`, `metrics`, `profile` and `profiling-download` routes (#6658). - `mc admin config get subnet` no longer prints the SUBNET license token or the cluster API key. `mc admin config export` still includes both, so an exported file continues to restore onto a licensed deployment (#6941). - `ServerInfo` no longer returns a hash derived from `MINIO_*` environment values. A new `admin:ConfigUpdate`-gated endpoint compares an expected hash and returns match status only, 204 for a match and 409 for a mismatch (#5178). ### Resource bounds Paths that sized an allocation or read a request body without a limit are now bounded: stored checksum metadata (#6715), Iceberg `_last_checkpoint` metadata, with a configurable ceiling defaulting to 100,000 parts (#6415, #6440), admin API request bodies (#4458), and unauthenticated STS request bodies (#5943). POST policy uploads reject an expired policy before reading the body, and accept a configurable clock skew for clients with stale signatures (#5073). Four crash paths reachable from an ordinary request are closed (#3987, #5133, #3925, #4879). ### Dependency updates - `golang.org/x/crypto` to v0.56.0, clearing GO-2026-6354 (CVE-2026-78662) and GO-2026-6355 (CVE-2026-56855), two SSH denial-of-service advisories, and GO-2026-6303 (CVE-2026-56854), an SSH `source-address` critical option not enforced for non-public-key auth callbacks. - `golang.org/x/mod` to v0.40.0, clearing GO-2026-6179 (CVE-2026-56865) and GO-2026-6180 (CVE-2026-56864). - `github.com/google/cel-go` to v0.30.0, clearing GO-2026-6094. - All modules moved to the Go 1.26.6 toolchain. --- ## New Features - **Memory bandwidth speedtest.** A new `POST /speedtest/mem` endpoint, surfaced through `madmin-go` `MemPerf()`, runs a saturating streaming copy on each node over per-thread buffers larger than cache and reports what the memory subsystem actually delivered, alongside the DIMM topology that figure should be judged against. Memory bandwidth caps large-object throughput on a many-core storage node and is the one resource no utilization counter shows — DMA from a drive or a NIC never touches a core, so CPU, disk and network can all read idle while DRAM is saturated. Nodes do not cooperate, so unlike the network speedtest this is meaningful on a single-node deployment. Topology capture records its own provenance: SMBIOS type 17 read directly from `/sys/firmware/dmi`, falling back to the EDAC tree, then to measured-only, so a consumer can tell an unreadable source from a measured zero (#6648). - **A budget for what parity upgrade may permanently cost an erasure set.** Parity upgrade is not a transient cost: healing restores the shards an object is missing but never returns it to the configured erasure coding, so an object written at 11+5 instead of 12+4 stays about 9% larger for its whole life. A drive that stays offline therefore costs a share of everything written while it is gone, and nothing bounded that. Each erasure set now has an `erasure.parity_upgrade_budget`, expressed as a percent of its capacity; the chance of upgrading falls as the budget is spent rather than stopping dead at a ceiling, spreading the reduced redundancy across the outage. A set seen whole starts again with the full budget, so the ceiling bounds one outage rather than rationing a set over its lifetime (#6834). - **`aistor` is reserved as a bucket name**, alongside `minio`, so a user bucket can never collide with the system namespace. A request to create a bucket named `aistor` is refused, the name is excluded from replication, and it is handled on all four addressing paths: the reserved-bucket request guard, the browser redirect prefixes, the virtual-host router matcher, and virtual-host bucket-name parsing. - **A configurable URI SAN scheme for certificate identity.** `identity_tls uri_san_scheme` (`MINIO_IDENTITY_TLS_URI_SAN_SCHEME`) sets which URI SAN scheme marks a client certificate as a workload identity, defaulting to `spiffe`. `AssumeRoleWithCertificate` previously recognized only `spiffe://`, so a deployment issuing SPIFFE-shaped identities under another scheme fell through to the Common Name and — since SVIDs usually leave the Subject empty — was rejected outright. Only the scheme is rewritten before validation, so identical trust-domain and path rules apply, and only the configured scheme is recognized (#6873). - **AIStor Tables roles are assigned automatically on `mc admin replicate add`.** The replica catalog scanner no longer has to be enabled by hand first. The site the command runs against stays the writable primary and its peer becomes the read-only replica, so argument order decides the roles. The server leaves both sites untouched when the topology is not exactly two sites, when either site is already a replica, or when the peer holds user warehouses of its own; if it cannot write the role the add still succeeds and reports the failure. Three guards protect the topology afterwards, refusing a promotion that would leave two writable sites or a topology that cannot hold a primary. User documentation for Tables replication ships with this release (#6617, #5539). - **`partNumber` is accepted on GET and HEAD in S3Express mode.** Three guards rejected it with `501 NotSupported` or `400 InvalidRequest`, though AWS documents the parameter for directory buckets without caveat. In AIStor each `AppendObject` lands as a new part, so addressing a part by number is meaningful there. The mutually exclusive `partNumber` + `Range` guard is preserved (#5304). - **Service accounts may call `AssumeRole` for another user.** The `roleArn` form of `AssumeRole` previously worked only for non-derived credentials, which excluded every external LDAP or OIDC user regardless of permissions. Derived credentials are now allowed on that path only; the ordinary `AssumeRole` path keeps its filter (#6898). - **`MINIO_RDMA_DIRECT_MAX_SIZE`** makes the crossover between a single direct RDMA write and the chunked multi-peer path tunable (default 64 MiB, unchanged) (#6859). --- ## Performance Improvements - **Erasure shards decode directly from the RDMA landing buffer.** A remote shard was RDMA-written into a pinned chunk and then copied into a stash buffer before erasure decode; the chunk is now aliased when the shard lies wholly inside it. Measured on a five-node cluster with inter-node RDMA: 22.6 to 26.5 GiB/s per node (+17%), `memmove` from 15.0% to 9.0% of CPU, and 40% less CPU per byte served (#6944). - **Both copies are removed from the inter-node RDMA PUT path** (#6951). - **Large RDMA transfers stay on the direct path, and the local node takes a share of chunk dispatch.** Chunk dispatch hashed over remote peers only, sending 100% of chunks off-box even though the coordinating node holds shards of the object itself; it now hashes over an extra slot meaning "serve locally" (#6859). - **Inter-node RDMA credit grants are proportional to consumption.** An even split is correct only under symmetric fan-in: a node writing more than its 1/N share consumed credits faster than it was granted them and ran a deficit that grew with bytes written, until every write blocked for the full credit wait timeout. On a six-node cluster with two nodes taking twice the S3 load, those two stalled after about 1.3 TB of inter-node writes; with proportional grants a full MLPerf llama3-70b checkpoint phase writes about 1.9 TB per node with zero errors and zero throttles. - **RDMA buffer registration is lazy per rail.** Registration previously touched every rail even though a transfer names one, so a caller registering per operation paid one registration per rail on every transfer. Register plus deregister of a 4 MiB buffer on a two-rail host went from 241.5 µs to 114.5 µs; a buffer reused across transfers still ends up registered everywhere, so failover is unchanged. - **Scanner heap on a wide folder no longer depends on how wide the folder is.** `scanFolder` collected every child into a worklist before scanning any of them, holding one joined path and one cache entry per child for the whole recursion — 3.5 GB per scanning drive on a field prefix holding about 34M objects. Past the forced-compaction threshold a compacted entry keeps no per-child cache state, so children are now scanned as they are read: at 4,000,000 children in one folder, heap drops from about 999 MiB to 107.7 MiB, and four times the children costs the same heap. Accounting, visit order and elapsed time are unchanged (#6903, #6913). - **Reclaiming abandoned temporary writes costs the same regardless of write volume.** The stale-upload sweep only ever inspected the top level of the staging directory, whose entries are touched by every new write, so abandoned writes aged out only at the next restart and a long-running server leaked them indefinitely. Writes now stage under a time-window directory, so the sweep can age a directory from its name with one `readdir` and reclaim it with a single rename. The delete trash moves to a sibling directory, so neither sweep can reach the other's data and the startup wipe of the staging area no longer removes pending deletes as a side effect (#6813). - **Replication status strings parse without a regexp.** These are a plain run of `arn=value;` pairs but had been parsed with a regexp since 2021, on a path that runs for every replicated version's metadata decode. With three targets: building the map went from 7990 ns/op and 9 allocations to 305 ns/op and 2, and a target lookup from 7560 ns/op and 7 allocations to 63 ns/op and none (#7094). - **A peer's AIStor Tables role no longer costs an argon2 derivation.** Reading it went through `GetConfigKV`, which decrypts the body with argon2id at 64 MiB a call, serialized on a package-global mutex. A customer heap profile attributed 548 GB of allocation — 4.9% of the total — to that path, about 6,200 calls per hour per node and 0.7 cores continuously, on a deployment with Tables disabled. The role is now read as plain JSON from the site-replication metadata response (#7015). - **Replication MRF queuing memory is bounded.** High backlog or latency on `GetObjectInfo` let decoded MRF entries accumulate in memory until the five-minute tick processed them; they are now bounded at one million entries, with overflow and duplicates dropped (#7013). - **An unsorted `max-keys=1` listing short-circuits into a STAT.** `listObjectsUnsorted` always ran a full namespace walk, so minfs-style directory probes paid multi-second walks — about 19 seconds observed — just to prove a leaf prefix is empty (#5958). - **Router performance improvements** from `minio/mux` v1.10.1 (#6801), zero-copy string-to-bytes conversion in `sipHashMod` and `crcHashMod` (#5018), and a `strings.NewReplacer` in the tracer's operation-name lookup (#5011). --- ## Bug Fixes ### Erasure coding, healing and data integrity - **An unversioned PUT with `If-None-Match: *` could return 503 forever on an object clients reported as absent.** A write-quorum miss during commit left two rival metadata groups on disk, and dangling detection keyed only by version ID, so it treated them as one group and never resolved the collision. Detection now keys on version ID and modification time, a heal retry is queued when a commit fails write quorum rather than waiting on the background scanner, and the collision is purged only when a drive is confirmed missing, neither group reaches quorum, every drive is accounted for, and no rival is a transitioned object (#7068). - **A shard aliased into the decode buffer is dropped before the next erasure block**, so it cannot be reused across blocks (#6972). - **Heal verification no longer reports a corrupt object as healthy.** The dry-run heal status was read from a single un-polled call, and an empty result list was treated as proof of health — but the server drains results on every read and reaches "finished" on a later poll than the one carrying them. Verification now polls the sequence to completion, accumulates every result for the object, and requires a confirmed-healthy verdict: no missing or corrupted shard on a reachable drive, no offline drive, and a non-empty post-heal drive list with every drive OK (#6766). - **A deleted object is no longer audited as unhealed.** A heal that found the version gone on every drive was logged as "unable to heal N missing blocks on drives", with both halves wrong: the object had been removed while the heal sat in the queue, and the message was synthesized from drive states over the real error. The reported error is now preserved, block counts move to their own tags, and a heal of a vanished object skips the audit entirely (#7146). - **Read-repair uses a deep scan when the trigger was bitrot corruption.** The repair path healed with the default scan mode, which validates part presence and size but never verifies bitrot checksums, so it could not detect the corruption that triggered it — losing heal-result accuracy and the per-drive silent-corruption counter (#6123). - **Heal tracker backups and the quarantine sweep are bounded.** Backup files accumulated one per heal restart for the life of the deployment; the newest three are now kept. The quarantine cleanup armed its 24-hour timer before its first iteration, so a node restarting more often than once a day never swept `.minio.sys/.quarantine` and never enforced `heal dangling_expiry`; the first sweep now runs a few minutes after startup (#6882). - **Heal configuration reads were unsynchronized.** All eight accessor methods on `heal.Config` had value receivers while `Update()` mutates in place under a mutex — the struct was copied at the call site before the read lock was ever taken, so the lock only serialized `Update()` against itself (#6889). - **The persisted heal-MRF file is bounded, and cordon-scoped goroutines are drained on shutdown.** The auto-heal, MRF and stale-multipart goroutines could still be touching state that shutdown had torn down (#6742). - **Heal tracker field writes are taken under the tracker mutex** rather than bare (#3804). - **Listing no longer fails with `NoSuchBucket` while a new pool settles.** After a cold restart that adds a pool, bucket directories are synced asynchronously, and the grid mesh can be fully established before that sync completes. Both the merged and unsorted listing paths returned the new pool's `errVolumeNotFound` immediately even though other pools had confirmed the bucket exists; that error is now non-fatal when at least one pool found the bucket (#5223). - **The cached bucket listing is scoped to its pool set.** A package-global cache bound its update function to the first object layer the process ever cached with, so a later pool set kept answering from the old one (#7166). - **Drive capacity reporting no longer breaks on an offline drive.** An offline or permission-denied drive reports size zero, producing a NaN percentage that `json.Marshal` refuses to serialize, which broke the drives query API (#3504). - **Drive-top queries filter by set rather than by pool.** A `SetIndex`-only filter seeded the metrics scope with the pool prefix, so it queried the wrong scope and reported every set drive offline on single-pool clusters (#6379). - **A transient decommission failure is retried before the run is marked failed.** Decommission turned any quorum blip — drives settling, a node reconnecting after a restart — into a terminal `Failed` status. Each attempt now resets its own failure counters and is judged only on failures it observes itself, so a resumed run no longer inherits its predecessor's count (#6447). - **The health check no longer floods critical-level logs.** Health runs on every liveness probe and metrics scrape, so a set below quorum logged on every call with no suppression. Suppression is now per pool and set, and the fatal log kind is dropped — it mapped write-quorum loss to syslog `Crit` while the strictly more severe read-quorum loss stayed at error (#6781). ### AIStor Tables, Delta Sharing and Iceberg - **Table metadata is no longer destroyed on an ambiguous pointer write.** Several catalog paths treated an error from a pointer write as proof that nothing was written and deleted on that basis — but the erasure commit is two-phase, and a commit below write quorum can still be served, so the cleanup deleted the metadata a live pointer named and lost the table. Cleanup now runs only on a precondition failure, the one error that carries proof. The same rule replaces two transaction-log deletions, both renames now CAS-guard the pointer write so a rename cannot silently overwrite a concurrent commit, and three rollbacks that ran on the request context — so a disconnected client skipped them — are detached and bounded (#6918). - **Delta Sharing stays consistent across concurrent table changes.** Share validation and the share write ran under separate locks, so a table dropped between the two left a share referencing a table that no longer exists. Referenced table-registry shard locks are now taken before the per-share mutation lock, matching the table-delete lock order, and the admin share list retries indexed shares missed by its fast unlocked scan so healthy shares no longer disappear during a rewrite (#7034). - **Two tables dropped at the same time could leave one referenced by a share.** The delete cascade relied on a best-effort unlocked scan of share files, and a share being rewritten during that scan was temporarily unreadable and silently skipped. The cascade now treats the share index as authoritative and re-checks missing or unreadable shares under their mutation lock (#6822). - **A failed Iceberg manifest read no longer returns HTTP 200.** Both manifest readers in Delta Sharing logged and continued on a fetch error, so a missing or undecryptable manifest produced an empty or partial file set returned as a success — which clients treat as the complete table, silently dropping rows (#6323). - **A lost compaction commit race is no longer recorded as a completed run** (#6777). - **The replica catalog scanner recovers from a missed configuration reload.** A node applies dynamic config at boot and on the admin-write peer fan-out, and that fan-out discards its per-peer errors while the API reports the change as applied — so a switched-off scanner could stay off for the node's lifetime. It now re-reads the stored settings once a minute, bounding a lost notice to about two minutes, re-arms its admin controls after recovering, and no longer restarts a rebuild retry's backoff on every pass (#6850). - **Maintenance gates are re-checked at every traversal boundary.** The snapshot-expiry gate was read once per maintenance cycle, and a cycle spans every warehouse in the cluster and can run for minutes — so an operator disabling snapshot expiry, or a promotion to replica mode, could watch snapshots keep being deleted for the rest of it. Snapshot expiry, unreferenced-file removal and compaction now re-read their gate at every warehouse, namespace and table boundary, including compaction's grid-distributed dispatch path (#6879). - **An empty namespace listing returns `[]` rather than `null`**, which pyiceberg and pydantic clients reject (#7005). - **A data race in the table registry shard singleflight** is closed by taking a read-only registry handle (#5895, #6161). - **Delete markers are no longer projected as multipart uploads in catalog inventory.** The multipart flag derived solely from ETag length, and a delete marker's ETag is empty. ### Replication and site replication - **Replication requests carry AIStor's upload limits rather than Amazon S3's.** A part or single PUT above the Amazon limits was refused locally, so an object AIStor had stored never reached the remote. Batch targets keep the Amazon limits unless the target is an AIStor peer, since only such a peer accepts a part above 5 GiB (#7145). - **A replicated DELETE is answered truthfully, so an upgraded peer needs no follow-up HEAD** (#7100). - **A hand-edited site-replication rule is now visible in `mc admin replicate status`.** The cross-site comparator ignores a disabled rule status on purpose, so healing cannot undo an operator's edit — but that also hid the edit from the status report. Separately, the comparator seated the first sorted site as an unchecked baseline, so a disabled setting on whichever deployment ID sorts first escaped both the status report and the bucket-replication healer (#7053). - **Site-replication rule priorities are no longer renumbered on every reconfiguration.** The handler assigned a fresh priority — highest existing plus ten — on every pass, including when the rule already existed unchanged. It runs on site expansion, peer endpoint updates, bucket heal and consistency-check mismatches, on every site, so values churned without bound (10/20 to 80/90 within one afternoon on a live three-site deployment) and surfaced as constant drift in `mc replicate ls` and exported configs. Because priorities are a validated uniqueness surface, a later rule edit passing `--priority` failed with "priority must be unique" once a reconfiguration round had renumbered them. Replication behavior itself is unchanged (#7052). - **Two unreachable peers during `EditPeerCluster` were a fatal concurrent map write.** Per-peer failures were recorded into a map shared by every fan-out goroutine; they now land in a slice indexed per edit. Two site-replication state reads that looked peers up without holding the read lock are also fixed. - **The pool lock is held across the replication worker channel send** (#6376). - **A torn-down target is reported as failed** rather than silently filtered in the notification queue's filtered send (#6764). ### Tiering - **Tier configuration recovers on demand instead of waiting for the next 15-minute refresh.** The config is loaded once at startup, and after a full-cluster restart that single read races the rest of startup — a peer still booting fails it and leaves the config unloaded. Every consumer that needs the config now attempts a singleflight reload, with concurrent callers collapsing onto one backend read, and returns `503 Service Unavailable` until it succeeds; a quorum blip on the S3 read path maps to 503 SlowDown. `Empty()`, `TierType()` and `ListTiers()` also recover on demand rather than reporting "no tiers" from empty startup state, bounded by a short two-second best-effort timeout because they sit on request paths (#6432, #6527). - **A failed startup tier-config load is retried on every deployment.** The background refresher started only on a distributed deployment, so on a single node a boot-time failure was logged once and the config stayed unloaded for the life of the process. Startup recovery now retries with jittered backoff on all deployments, reading under the full reload timeout so a slow-but-working backend is still recovered (#6653). - **A tier save uses its receiver rather than the global tier manager** (#4048). ### IAM, STS and authentication - **Two OpenID providers sharing an issuer no longer silently drop one.** The issuer-to-provider map held a single provider per issuer, so configuring two providers against the same IDP with distinct client IDs discarded one (#5115). - **OIDC token rejections return the AWS error codes.** An `exp` claim readable as a past timestamp now answers `ExpiredToken`, and every other web-identity token rejection answers a new `InvalidIdentityToken`; both previously came back as the generic `InvalidParameterValue`. The expiry verdict is reached before the JWKS refresh retry, so the code stays correct while an IdP's JWKS endpoint is unreachable, and one pointless JWKS fetch per rejected token is removed (#6845). - **A malformed `identity_tls` value is rejected when it is set, and tolerated when it is loaded.** Validation ran only after reading the enable flag and with environment lookups off, so a bad `uri_san_scheme`, `skip_verify` or `spiffe_multi_domain` value was stored unchecked and then took the X.509/TLS STS API down on the next start. Values are now parsed before the flag, and the load path substitutes the key's default with a warning rather than failing (#7008). - **`kms:KeyStatus` is accepted on the admin KMS status handlers**, which gated on `admin:KMSKeyStatus` alone (#6553). - **`AccountStatus` is populated in bulk access-key listings.** The OpenID bulk handler dropped it on both STS keys and service accounts, and the builtin bulk handler dropped it on STS keys, so clients could not tell which keys were disabled without an extra info call per key (#5469). - **`ParentUserStatus` is reported correctly for external parents.** A service account or STS credential whose parent is an LDAP or OIDC user always reported as disabled, though no mechanism exists to disable such a user. The field is also no longer carried on the credential structure, and is filled in by the `InfoServiceAccount` and `InfoAccessKey` handlers that are its only readers (#6895). - **`ListAccessKeysBulk` works for any user calling it on themselves**, mirroring the other builtin access-key APIs (#5411). - **`InfoAccessKey` looks up the requested credential's key**, not the caller's, when resolving a service account (#4673). - **IAM list methods report context cancellation** by returning the context error (#4625). - **A user's effective `UpdatedAt` is computed from all related entities**, not just the user record (#3759). - **A token revoke claim is no longer added when revocation is disabled** (#4822). - **Inventory job IAM checks moved to the handler.** The runtime job runner resolved credentials through the IAM store to evaluate permissions on each execution, but LDAP users and STS tokens are ephemeral and not stored there, so jobs errored at runtime even when the submitting user had valid permissions. A caller without permission now gets `AccessDenied` at config creation (#4620). - **Batch job metrics use a fine-grained permission.** Requesting only batch-job metrics required `admin:ServerInfo`; it now accepts `admin:DescribeBatchJob`, so a user with minimal batch permissions can check job status without being granted broad monitoring access (#4662). - **A context that is already done unblocks `once.DoWithContext`** instead of pinning the caller's goroutine behind a slow in-flight initialization (#5129). ### S3 API and compatibility - **`CompleteMultipartUpload` emits `"` rather than `"` for the ETag.** Go's XML encoder writes the decimal entity, but AWS CRT decodes only the named one when stripping ETag quoting before sending `If-Match` — so CRT-based clients such as Mountpoint and AWS CLI v2 failed conditional requests with HTTP 412 after completing a multipart upload (#4590). - **A bucket SSE-KMS rule may omit `KMSMasterKeyID`.** `PutBucketEncryption` rejected an `aws:kms` rule without an explicit key ID, while AWS S3 accepts one and falls back to its managed key. The rule is now accepted and objects are encrypted with whichever key the KMS holds as its default at write time. The default key is resolved when the configuration is set, so a missing or unreachable key fails `PutBucketEncryption` rather than every subsequent upload. Under site replication the configuration propagates to peer sites, and a peer still on an earlier release cannot read a rule stored in this form — set an explicit key ID until every site carries this release (#6794). - **A request in flight during a node shutdown answers 503 rather than being audited as a client disconnect.** Shutdown cancelled the global context with no cause, so such requests recorded a 499 even though the client was still connected and would receive the response. They now carry a retriable `503 XMinioServerShuttingDown` with a `Retry-After` header, which S3 clients retry automatically. A genuine client-initiated disconnect still records 499 (#4402). - **A malformed `partNumber` or `x-minio-force-delete` value answers `400 InvalidArgument`.** A non-integer part number returned `500 InternalError` with raw parser text in the body, and a zero or negative one returned an admin-API error code on the S3 data path; an unparseable force-delete header returned `500 InternalError`. Every value previously accepted for the header is still accepted (#7010). ### Scanner, ILM, inventory and batch - **Negative `alert_excess_versions` and `alert_excess_folders` values are rejected.** Both were parsed without bounds, so a `-100` typo for `100` was stored verbatim — and because these are thresholds in `count > threshold` comparisons on the object hot path and in the scanner, a negative threshold made every comparison true and fired excess events for every object and folder, flooding the alert subsystem (#6076). - **`NumVersions` is populated at write time for many-versions checks** (#6652). - **Each ILM transition action is counted exactly once.** The transition counter incremented on every non-error return, including the already-transitioned no-op path, so when a write-path immediate transition raced a background ILM sweep the second call no-opped but was still counted, inflating the metric above the number of objects actually transitioned (#6232). - **Bucket scan state is cleared before the anti-stampede delay** (#4928). - **Inventory counting metrics reset when a new executor recovers an expired lock.** A second executor taking over a job whose lock had expired accumulated on top of the previous holder's saved counts, double-counting scanned objects (#4245). - **Expected inventory schedule contention is no longer logged as an error** (#6284). - **Legacy pre-2025 batch job report IDs age out in cleanup** (#6628). - **A batch job refused at `Start` is recorded as failed.** The refusal was logged and the job never retried, but the persisted record kept `Failed=false`, so `batch status` reported a job that would never run again as still running (#6808). - **Batch configurations can be edited again** (#4104). - **Windowed metrics anchor a whole-window read at the active segment**, and a write into a past slot updates the window start index (#6563, #5505). ### Configuration, admin and observability - **Two concurrent configuration updates could drop each other's edit.** Every writer rewrites the whole configuration, so a single-key update and a bulk import could each overwrite the other. The read-modify-write in both the single-key path and the bulk import now runs under the server configuration lock, held across the read and the save only. - **`enable` status is always present in a subsystem config read.** `mc admin config get ` and the bulk config export omitted the `enable` key when its value matched the subsystem default of `on`, leaving every client to reconstruct enabled state from help metadata. An explicit `enable=on` / `enable=off` row is now always emitted (#4716). - **Scanner excess-folder and excess-version detections are counted rather than logged.** The per-detection log and audit entries are replaced by two Prometheus counters on the `/scanner` endpoint, `minio_scanner_excess_folders_total` and `minio_scanner_excess_versions_total`, which gives operators a countable signal instead of log volume. Event notifications and the admin API's `excessPaths` tracking are unchanged (#6861). - **A port on a later multi-NIC address now produces a startup warning.** The server accepts such a port, strips it, and sends all inter-node traffic to the port of the first address — until now with no signal to the operator (#7076). - **Negative network timeout durations are rejected at config load.** `idleTimeout`, `readHeaderTimeout`, `connUserTimeout` and `dnsCacheTTL` accepted any value the duration parser accepted, including negatives, which were then silently discarded by positive-value guards at the consumption sites — so the configured value was ignored and the default used with no feedback (#6077). - **An unknown key on an external notification target logs a warning instead of failing the whole notification subsystem**, and a validation error now names only the rejected keys rather than their values (#3836). - **QoS metrics count refusals.** The throttled counter published every request a rule matched, admitted or refused, under a help string that says throttled, while the value rendered by `mc qos status` as "Throttled requests" counted the requests the rule let through. It now counts refusals and equals the sum of the three reason counters. `POST ?restore` mapped to an invalid rule name, so restores could not be throttled at all, and every bucket subresource DELETE — `?tagging`, `?policy`, `?lifecycle`, `?encryption`, `?replication`, `?website`, `?cors` — collapsed to `s3.DeleteBucket`, so each specific rule was inert while a `s3.DeleteBucket` rule silently throttled subresource deletes. Refusals now also reach the audit log (#6896). - **QoS rate-limit 429s appear in `mc admin trace`** on the S3 trace rather than the internal one (#6144). - **Admin error messages no longer duplicate their text.** Errors with an explicit API code mapping had the raw underlying error appended to an already-authoritative description (#4078). - **IAM and bucket-metadata zip exports carry as much data as can be read** rather than failing wholesale (#4821). - **A data race in the logger's sampling decision** is closed; the shared RNG was not safe to share (#6354). - **Log records are no longer dropped between a flush signal and the flush.** The recorder sized its buffers at exactly the flush trigger, and the flush is signalled through a non-blocking send — so every record written in between overflowed capacity and evicted the oldest entry. Raising `flush_count` at runtime made it strictly worse by lifting the trigger without resizing. Buffers are now sized above the trigger and resized on a runtime change. - **An unreachable peer no longer produces an error line on every refresh interval.** The IAM and Tables log helpers filtered a disconnect inconsistently or not at all; both now drop it, the noisiest call sites use per-target suppression, and repeat summaries gain an hourly floor so a condition's log volume tracks how long it has been failing rather than how fast the caller retries (#6795). - **Log deduplication keys and context passed to the logger are no longer discarded**, and dropped error context is folded into the logged error (#6745, #6746). - **Log suppression IDs are scoped to the entity that failed**, so a failure on one drive, bucket or peer no longer suppresses the same diagnostic for every other one (#6655, #6659, #6694). - **A client cancelling a request no longer emits canceled-context error logs** from the metadata walk (#4432). - **Grid error matching handles the pointer form, and a disconnect is no longer misread as a remote error.** `(*RemoteErr).Is` compared pointer addresses rather than the error values they contained, so the pointer form never matched under `errors.Is`. `ErrDisconnected` is now a plain sentinel rather than a remote error — a disconnect is local transport information, meaning this node could not reach the peer, not that the peer handled the request and returned an error — and fan-out gains an error-kind field so an unreachable peer stays distinguishable from a remote error carrying the same text (#6838, #6900). - **A peer running without a configuration file no longer panics the grid encoder** mid-RPC; the handler answers with an encodable marker that the consistency validator reads as missing configuration rather than a checksum divergence (#4692). - **A hotfix upgrade to a binary with the same release tag but a different commit now proceeds.** Verification compared release times only, so it rejected the update as "already running the latest version"; it now compares binary checksums when release times match (#3934). - **The docs URLs the server prints point at the current base path.** Eleven runtime-visible occurrences — the startup banner, the `mc` quickstart link, the missing-certificate hints, the one-at-a-time upgrade warning, the Kubernetes deployment pointer, the erasure-coding and TLS config errors, and the post-install instructions — used a path that redirects rather than 404s, so nothing looked broken while operators were handed a stale URL (#6772). - **Legacy Console updated to v0.0.46**, refreshing its dependencies (#7041). ### RDMA - **The inter-node RDMA inflight window is floored at one write chunk.** The window's ceiling equalled the chunk size and its floor was well below it, so one chunk saturated the whole window and a multiplicative decrease on a congestion signal walked it below one chunk. Nothing is then admissible, and the window only widens on a clean completion, which can no longer happen — the sender blocked for the full 60-second credit wait, and because a stalled write holds one of 32 per-peer slots for that whole time, PUTs stalled cluster-wide while the error count stopped growing. The clamp runs after the environment overrides, so the tuning variables cannot reintroduce the deadlock, and the defaults are now 8 MiB initial, 4 MiB floor and 32 MiB ceiling (#6992). - **The RDMA device is resolved by name, so GPU-Direct works on InfiniBand** (#6828). - **The peer LID is put on the wire, so inter-node RDMA works on InfiniBand.** - **The vendored transport headers are re-synced with their libraries.** The libraries had been refreshed without the headers beside them, so cgo compiled against declarations one revision behind the object it linked. That was not cosmetic: the peer endpoint struct had grown by four bytes, so Go laid out the endpoint array at the old stride while the library read it at the new one, and every element after the first came back with a wrong GID. A single-NIC host has one element and looks healthy; a dual-NIC host does not. - **Every rail is bound when no address pins the server to one.** - **IPv6 fabrics are supported.** Both transports parsed a bind address as IPv4 only and matched the GID table against it, so an IPv6-only fabric failed at init with "no RDMA device found for IP". Measured at 400 Gb/s NDR on a single rail, IPv6 runs within 0.5% of IPv4 — the difference is the larger header. Two operator-visible consequences: `0.0.0.0` or `::` as an RDMA bind address used to fail and now means every device with an active port, and rail selection matches a native IPv6 peer on its /64, so a dual-rail host stops round-robining onto a rail with no route. Note that mlx5 synthesizes a MAC-derived link-local GID whether or not IPv6 is configured on the host, so an address can resolve and a queue pair come up while moving zero bytes: `net.ipv6.conf.all.disable_ipv6` must be 0, or there is no neighbour discovery to resolve the destination MAC with (#6923). - **A client that walks away mid-PUT no longer inflates the RDMA error metric.** Every write still in flight is cancelled, and each was logged at error level and counted against the peer — one autotuning `mc support perf object` run produced 3,660 such lines across six nodes, burying the fabric failures the logging exists to surface. An expired deadline is still reported: that is what a stalled peer looks like. --- ## Upgrade Instructions For detailed upgrade instructions, please read: https://docs.min.io/aistor/upgrade-aistor-server/ Platform-specific upgrade guides: - **Linux/Bare Metal**: https://docs.min.io/aistor/upgrade-aistor-server/upgrade-aistor-linux/ - **Kubernetes with Helm**: https://docs.min.io/aistor/upgrade-aistor-server/upgrade-aistor-kubernetes-helm/ ### New Configuration Options | Setting | Default | Purpose | | ---------------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------- | | `erasure parity_upgrade_budget` / `MINIO_ERASURE_PARITY_UPGRADE_BUDGET` | `1.0` | Percent of an erasure set's capacity that parity upgrade may permanently consume | | `identity_tls uri_san_scheme` / `MINIO_IDENTITY_TLS_URI_SAN_SCHEME` | `spiffe` | URI SAN scheme that marks a client certificate as a workload identity | | `api post_policy_skew` / `MINIO_API_POST_POLICY_SKEW` | `0s` | Tolerance for a POST policy expiration in the past | | `MINIO_RDMA_DIRECT_MAX_SIZE` | 64 MiB | Crossover between a single direct RDMA write and the chunked multi-peer path | New Prometheus counters on the metrics `/scanner` endpoint: `minio_scanner_excess_folders_total` and `minio_scanner_excess_versions_total`, replacing the per-detection excess-folder and excess-version log and audit entries. Three things to check before upgrading, none of which apply to a default deployment: - **A bucket named `aistor`**, if one exists, since the name is now reserved. - **Alerting rules built on the scanner excess-folder or excess-version log lines**, which should read the new `minio_scanner_excess_folders_total` and `minio_scanner_excess_versions_total` counters instead. - **A site-replication topology whose sites are not all on this release**, where a bucket SSE-KMS rule should keep an explicit `KMSMasterKeyID` until every site carries this release. ### Support For enterprise support: - SUBNET Support: https://subnet.min.io - Documentation: https://docs.min.io