Xet documentation
CAS API Documentation
CAS API Documentation
This document describes the HTTP API endpoints used by the Content Addressable Storage (CAS) client to interact with the remote CAS server.
Authentication
To authenticate, authorize, and obtain the API base URL, follow the instructions in Authentication.
Converting Hashes to Strings
Sometimes hashes are used in API paths as hexadecimal strings (reconstruction, xorb upload, global dedupe API).
To convert a 32 hash to a 64 hexadecimal character string to be used as part of an API path there is a specific procedure, MUST NOT directly convert each byte.
Procedure
For every 8 bytes in the hash (indices 0-7, 8-15, 16-23, 24-31) reverse the order of each byte in those regions then concatenate the regions back in order.
Otherwise stated, consider each 8 byte part of a hash as a little endian 64 bit unsigned integer, then concatenate the hexadecimal representation of the 4 numbers in order (each padded with 0’s to 16 characters).
In all cases that a hash is represented as a string it is converted from a byte array to a string using this procedure.
Example
Suppose a hash value is: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
Then before converting to a string it will first have its bytes reordered to: [7, 6, 5, 4, 3, 2, 1, 0, 15, 14, 13, 12, 11, 10, 9, 8, 23, 22, 21, 20, 19, 18, 17, 16, 31, 30, 29, 28, 27, 26, 25, 24]
So the string value of the provided hash [0..32] is NOT 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f.
It is: 07060504030201000f0e0d0c0b0a090817161514131211101f1e1d1c1b1a1918.
Endpoints
1. Get File Reconstruction
This v2 endpoint is the recommended default. Because older CAS deployments may not serve
/v2/, clients SHOULD fall back to the deprecatedGET /v1/reconstructions/{file_id}on a404or501.
Description: Retrieves reconstruction information for a specific file. Returns URLs optimized for multi-range fetching: multiple byte ranges for the same xorb are combined into a single URL. Supports byte range via the optional
Rangeheader.Path:
/v2/reconstructions/{file_id}Method:
GETParameters:
file_id: File hash in hex format (64 lowercase hexadecimal characters). See file hashes for computing the file hash and converting hashes to strings.
Headers:
Range: OPTIONAL. Format:bytes={start}-{end}(end is inclusive).Accept-Encoding: OPTIONAL. The server supportsgzipandzstdcompression on the JSON response. Clients SHOULD sendAccept-Encoding: gziporAccept-Encoding: zstdto reduce reconstruction response size.
Minimum Token Scope:
readBody: None.
Response: JSON (
QueryReconstructionResponse){ "offset_into_first_range": 0, "terms": [...], "xorbs": {...} }Error Responses: See Error Cases
400 Bad Request: Malformedfile_idin the path. Fix the path before retrying.401 Unauthorized: Refresh the token to continue making requests, or provide a token in theAuthorizationheader.404 Not Found: The file does not exist. Not retryable.416 Range Not Satisfiable: The requested byte range start exceeds the end of the file. Not retryable.
GET /v2/reconstructions/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef -H "Authorization: Bearer <token>" OPTIONAL: -H Range: "bytes=0-100000"
Example File Reconstruction Response Body
See QueryReconstructionResponse for more details in the download protocol specification.
2. Query Chunk Deduplication (Global Deduplication)
- Description: Checks if a chunk exists in the CAS for deduplication purposes.
- Path:
/v1/chunks/{prefix}/{hash} - Method:
GET - Parameters:
prefix: The only acceptable prefix for the Global Deduplication API isdefault-merkledb.hash: Chunk hash in hex format (64 lowercase hexadecimal characters). See Chunk Hashes to compute the chunk hash and converting hashes to strings.
- Minimum Token Scope:
read - Body: None.
- Response: Shard format bytes (
application/octet-stream), deserialize as a shard. - Error Responses: See Error Cases
400 Bad Request: Malformed hash in the path. Fix the path before retrying.401 Unauthorized: Refresh the token to continue making requests, or provide a token in theAuthorizationheader.404 Not Found: Chunk not already tracked by global deduplication. Not retryable.
GET /v1/chunks/default-merkledb/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef -H "Authorization: Bearer <token>"
Example Shard Response Body
An example shard response body can be found in Xet reference files.
3. Upload Xorb
- Description: Uploads a serialized Xorb to the server; uploading real data in serialized format.
- Path:
/v1/xorbs/{prefix}/{hash} - Method:
POST - Parameters:
prefix: The only acceptable prefix for the Xorb upload API isdefault.hash: Xorb hash in hex format (64 lowercase hexadecimal characters). See Xorb Hashes to compute the hash, and converting hashes to strings.
- Headers:
Content-Length: REQUIRED. Size of the serialized Xorb in bytes. Requests without a validContent-Lengthare rejected with400 Bad Request. Most HTTP clients set it automatically for a fixed-size body; set it explicitly when streaming the body.
- Minimum Token Scope:
write - Body: Serialized Xorb bytes (
application/octet-stream). See xorb format serialization. - Response: JSON (
UploadXorbResponse)
{
"was_inserted": true
}- Note:
was_insertedisfalseif the Xorb already exists; this is not an error.
Clients MAY instead upload the Xorb directly to object storage through an upload grant. This endpoint remains available, and clients MUST use it when the grant request returns
404 Not Found.
- Error Responses: See Error Cases
400 Bad Request: Missing or invalidContent-Lengthheader, malformed hash in the path, Xorb hash does not match the body, or body is incorrectly serialized.401 Unauthorized: The token is missing, invalid, or expired, or does not havewritescope (for example, areadtoken). Refresh the token, or provide awritescope token.
POST /v1/xorbs/default/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef -H "Authorization: Bearer <token>"
Example Xorb Request Body
An example xorb request body can be found in Xet reference files.
4. Upload Shard
- Description: Uploads a Shard to the CAS. Uploads file reconstructions and new xorb listing, serialized into the shard format; marks the files as uploaded.
- Path:
/v1/shards - Method:
POST - Headers:
Content-Length: REQUIRED. Size of the serialized Shard in bytes. Requests without a validContent-Lengthare rejected with400 Bad Request. Most HTTP clients set it automatically for a fixed-size body; set it explicitly when streaming the body.
- Minimum Token Scope:
write - Body: Serialized Shard data as bytes (
application/octet-stream). See Shard format guide. - Response: JSON (
UploadShardResponse)
{
"result": 1,
"shard_hash": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
}- Where
resultis:0: The Shard already exists.1:SyncPerformed— the Shard was registered.
shard_hash: the hash the server stored the Shard under, in the hash string format. The server re-serializes the Shard before storing it, so this can differ from a hash the client computed over the uploaded bytes. Older servers omit this field.
The value of result does not carry any meaning, if the upload shard API returns a 200 OK status code, the upload was successful and the files listed are considered uploaded.
- Error Responses: See Error Cases
400 Bad Request: Missing or invalidContent-Lengthheader, Shard is incorrectly serialized, or Shard contents failed verification.- Can mean that a referenced Xorb doesn’t exist or the shard is too large
401 Unauthorized: The token is missing, invalid, or expired, or does not havewritescope (for example, areadtoken). Refresh the token, or provide awritescope token.
POST /v1/shards -H "Authorization: Bearer <token>"
Example Shard Request Body
An example shard request body can be found in Xet reference files.
5. Streaming Shard Upload
- Description: Runs the same validation and registration as Upload Shard, but streams finalization progress back as newline-delimited JSON so clients can show real progress on large shards, which can take several seconds to finalize server-side.
- Path:
/v2/shards - Method:
POST - Headers:
Content-Length: REQUIRED. Size of the serialized Shard in bytes. Requests without a validContent-Lengthare rejected with400 Bad Request.
- Minimum Token Scope:
write - Body: Serialized Shard data as bytes (
application/octet-stream), identical to/v1/shards. See Shard format guide. - Response: A newline-delimited JSON stream (
Content-Type: application/x-ndjson,Cache-Control: no-cache), one JSON object (“event”) per line.
The server starts the response only after it has received the full request body, so a failure is reported only after the whole body is sent.
The HTTP status is then 200 OK, so success or failure is carried by the terminal event, NOT by the status code.
Clients MUST read the stream to completion and inspect the terminal event.
Verification starts while the body is still arriving, but progress from that period is not streamed: when the response starts, the server sends the most recent event, then streams events as they happen. The first event can therefore be any event, including the terminal one.
Each event has a type field. Clients SHOULD ignore events with an unknown type.
validating: verification progress.verifiedis the number of completed verification tasks,totalthe number spawned so far.totalcan still grow, so treatverified / totalas a live ratio, not a final percentage.{"type":"validating","verified":120,"total":512}committing: the shard is being durably written.stageidentifies the sub-step, so a stalled commit shows where it is stuck:uploading: persisting the shard object to storage.syncing: registering the shard (file ids, global dedup, shard list).
{"type":"committing","stage":"uploading"}result: terminal success; aresultevent means the upload succeeded. It carries no other fields: unlike/v1/shards, the stream does not report whether the Shard already existed, or the stored Shard hash.{"type":"result"}error: terminal failure with a sanitizedmessageand aretryableflag. Because the status is already200 OK, this event is the only failure signal once streaming has started. Ifretryableistrue(a transient server-side fault), the client SHOULD retry the upload. If it isfalse(for example, the Shard failed verification), the client MUST NOT retry the same Shard.{"type":"error","message":"...","retryable":false}
A typical successful stream:
{"type":"validating","verified":256,"total":512}
{"type":"validating","verified":512,"total":512}
{"type":"committing","stage":"uploading"}
{"type":"committing","stage":"syncing"}
{"type":"result"}If validation stalls, the server periodically re-emits the last progress event as a heartbeat so the stream never looks dead, which lets clients keep a short read timeout.
Additive change:
/v1/shardsis unchanged. Clients SHOULD try/v2/shardsand fall back to/v1/shardson404 Not Foundor501 Not Implemented.
- Error Responses: A malformed or unauthorized request fails with a status code before the stream starts; see Error Cases. Once the stream has started (
200 OK), validation and registration failures are reported through the terminalerrorevent instead.400 Bad Request: Missing or invalidContent-Lengthheader.401 Unauthorized: The token is missing, invalid, or expired, or does not havewritescope (for example, areadtoken). Refresh the token, or provide awritescope token.
POST /v2/shards -H "Authorization: Bearer <token>"
6. Batch File Reconstruction
Description: Retrieves reconstruction information for multiple files in a single request, avoiding N separate calls.
Path:
/v1/reconstructionsMethod:
POSTHeaders:
Accept-Encoding: OPTIONAL. The server supportsgzipandzstdcompression on the JSON response. Clients SHOULD sendAccept-Encoding: gziporAccept-Encoding: zstdto reduce response size.
Minimum Token Scope:
readBody: JSON array of keys, each a
{ "prefix", "hash" }object.prefixisdefault;hashis the file hash in hex format (64 lowercase hexadecimal characters). Duplicate keys are de-duplicated server-side.[ { "prefix": "default", "hash": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" } ]Response: JSON (
BatchQueryReconstructionResponse){ "files": { "0123...": [/* CASReconstructionTerm, ... */] }, "fetch_info": { "0123...": [/* CASReconstructionFetchInfo, ... */] } }filesmaps each file id to its reconstruction terms;fetch_infois shared across files and keyed by xorb hash. This endpoint returns the v1 response shape; for the multi-range optimized format, call/v2/reconstructions/{file_id}per file.Error Responses: See Error Cases
400 Bad Request: Malformed body or invalid file id in the request.401 Unauthorized: Refresh the token to continue making requests, or provide a token in theAuthorizationheader.404 Not Found: One or more files do not exist. Not retryable.
POST /v1/reconstructions -H "Authorization: Bearer <token>" -H "Content-Type: application/json"
7. Head Xorb
- Description: Existence check for a Xorb.
- Path:
/v1/xorbs/{prefix}/{hash} - Method:
HEAD - Parameters:
prefix: The only acceptable prefix for Xorbs isdefault.hash: Xorb hash in hex format (64 lowercase hexadecimal characters). See Xorb Hashes and converting hashes to strings.
- Minimum Token Scope:
read - Body: None.
- Response: Empty body. Response headers:
Content-Length: size in bytes of the stored Xorb.
- Error Responses: See Error Cases
400 Bad Request: Malformed hash in the path.401 Unauthorized: Refresh the token to continue making requests, or provide a token in theAuthorizationheader.404 Not Found: The Xorb does not exist. Not retryable.
HEAD /v1/xorbs/default/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef -H "Authorization: Bearer <token>"
8. Head File
- Description: Existence check for a file.
- Path:
/v1/files/{file_id} - Method:
HEAD - Parameters:
file_id: File hash in hex format (64 lowercase hexadecimal characters). See file hashes and converting hashes to strings.
- Minimum Token Scope:
read - Body: None.
- Response: Empty body. Response headers:
Content-Length: size in bytes of the full file.
- Error Responses: See Error Cases
400 Bad Request: Malformedfile_idin the path.401 Unauthorized: Refresh the token to continue making requests, or provide a token in theAuthorizationheader.404 Not Found: The file does not exist. Not retryable.
HEAD /v1/files/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef -H "Authorization: Bearer <token>"
9. Get File Chunk Hashes
Description: For a file and a set of “dirty” byte ranges (regions the client intends to re-chunk), returns the chunk windows the client must re-chunk plus opaque hash subtrees covering the unchanged gaps. Used by delta uploads to compose a new shard without re-uploading already-known chunks. The response covers the full file.
Path:
/v2/file-chunk-hashes/{file_id}Method:
GETParameters:
file_id: File hash in hex format (64 lowercase hexadecimal characters). See file hashes and converting hashes to strings.
Headers:
X-Range-Dirty: REQUIRED. One or more byte ranges using the standardbytes=syntax (inclusive ends, comma-separated), e.g.bytes=0-1023,5000-9999. Ranges are sorted and merged server-side. At least one range MUST be present.Accept-Encoding: OPTIONAL. The server supportsgzipandzstdcompression on the JSON response.
Minimum Token Scope:
readBody: None.
Response: JSON (
FileChunkHashesResponse). Unlike other endpoints in this spec, this response usescamelCasefield names, matching the server.{ "totalChunks": 1234, "fileSize": 9876543, "windows": [ { "dirtyByteRange": [0, 65536] } ], "hashRanges": [ null, "<opaque MerkleHashSubtree>" ], "gapVerification": [ "<hex MerkleHash>" ] }windows: one entry per dirty range (adjacent dirty ranges that share the same chunk are merged). EachdirtyByteRangeis[start, end)with an exclusive end (note: theX-Range-Dirtyrequest header uses inclusive ends). Bounds are chunk-aligned, expanded outward to fully contain the requested dirty range; the client re-chunks this exact span.hashRanges:N + 1entries forNwindows —[before_w0, between_w0_w1, ..., after_wN]. Each entry is an opaqueMerkleHashSubtree; pass as-is to the client merge routine.nullwhen the gap is empty (for example, a window starts at chunk 0).gapVerification: one hash per stable original segment (a segment lying entirely in a gap between dirty windows or before/after them, in segment order). The client wraps each into aFileVerificationEntryfor the verification section of the composed shard. Empty when every segment overlaps a dirty range.
Error Responses: See Error Cases
400 Bad Request: MissingX-Range-Dirtyheader, or malformedfile_id.416 Range Not Satisfiable: MalformedX-Range-Dirtyvalue (including open-ended ranges such asbytes=100-), or no dirty range overlaps the file.401 Unauthorized: Refresh the token to continue making requests, or provide a token in theAuthorizationheader.404 Not Found: The file does not exist. Not retryable.
GET /v2/file-chunk-hashes/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef -H "Authorization: Bearer <token>" -H "X-Range-Dirty: bytes=0-65535,200000-299999"
10. Get File Reconstruction (v1)
Deprecated. Use
GET /v2/reconstructions/{file_id}instead. This v1 endpoint is still served for existing clients but will be removed once they migrate. The v2 endpoint returns the multi-range optimized response described in section 1.
Description: Retrieves reconstruction information for a specific file, includes byte range support when
Rangeheader is set. Returns the v1 response shape (terms+fetch_info).Path:
/v1/reconstructions/{file_id}Method:
GETParameters:
file_id: File hash in hex format (64 lowercase hexadecimal characters). See file hashes for computing the file hash and converting hashes to strings.
Headers:
Range: OPTIONAL. Format:bytes={start}-{end}(end is inclusive).
Minimum Token Scope:
readBody: None.
Response: JSON (
QueryReconstructionResponse){ "offset_into_first_range": 0, "terms": [...], "fetch_info": {...} }Error Responses: See Error Cases
400 Bad Request: Malformedfile_idin the path. Fix the path before retrying.401 Unauthorized: Refresh the token to continue making requests, or provide a token in theAuthorizationheader.404 Not Found: The file does not exist. Not retryable.416 Range Not Satisfiable: The requested byte range start exceeds the end of the file. Not retryable.
GET /v1/reconstructions/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef -H "Authorization: Bearer <token>" OPTIONAL: -H Range: "bytes=0-100000"
11. Request Xorb Upload Grant
Description: Requests a grant to upload a serialized Xorb directly to object storage instead of through the CAS server. After uploading to the grant, the client MUST call Commit Xorb Upload. This flow is OPTIONAL: Upload Xorb always works, and clients MUST fall back to it when this endpoint returns
404 Not Found.Path:
/v1/xorb-grants/{prefix}/{hash}Method:
POSTParameters:
prefix:default.hash: Xorb hash in hex format (64 lowercase hexadecimal characters). See Xorb Hashes and converting hashes to strings.
Headers:
Content-Type:application/json.
Minimum Token Scope:
writeBody: JSON (
XorbUploadGrantRequest){ "length": 1234567, "checksum": { "algo": "crc64nvme", "value": "18446744073709551615" } }length: size in bytes of the serialized Xorb. MUST be greater than 0.checksum: CRC-64/NVME checksum of the serialized Xorb bytes.valueis the unsigned 64-bit checksum written as a decimal string, since JSON numbers lose precision above 2^53.crc64nvmeis the only supportedalgo.
Response:
201 Created: an upload is needed. JSON (XorbUploadGrant):{ "id": "01JBX6Q0Z8K4T3V9M2N5P7R1SA", "method": "PUT", "url": "https://...", "headers": { "If-None-Match": "*", "...": "..." } }id: opaque grant id, used to commit the upload.method,url,headers: send the serialized Xorb as the body of amethodrequest tourl, with exactly these headers.urlis a short-lived pre-signed URL on an object storage host, not on the CAS server; treat it as opaque. Do NOT send the CASAuthorizationheader tourl.
200 OKwith an empty body: the Xorb already exists. Nothing needs to be uploaded or committed.
Error Responses: See Error Cases
400 Bad Request: Malformed JSON, malformed hash in the path, orlengthis 0 or larger than the maximum Xorb size.401 Unauthorized: The token is missing, invalid, or expired, or does not havewritescope (for example, areadtoken). Refresh the token, or provide awritescope token.404 Not Found: The server does not offer an upload grant for this request (servers that predate this API also return404). This is not an error: upload throughPOST /v1/xorbsinstead.422 Unprocessable Entity: The body is valid JSON, butchecksumis missing or uses an unknownalgo.
Uploading to the Grant URL
The response to the upload request sent to the grant url determines the next step:
2xx: the Xorb was uploaded; call Commit Xorb Upload.412 Precondition Failed: an earlier attempt with this grant already stored the data; treat it as success and commit.409 Conflict: an earlier attempt with this grant is still being stored; retry with backoff.400 Bad Requestor403 Forbidden(for example, the grant expired): request a new grant and upload again. Clients SHOULD limit the number of new grants per Xorb (the reference client requests up to 3), then upload throughPOST /v1/xorbs.- Any other failure (for example, the object storage host cannot be reached): upload through
POST /v1/xorbs.
POST /v1/xorb-grants/default/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef -H "Authorization: Bearer <token>" -H "Content-Type: application/json"
12. Commit Xorb Upload
- Description: Validates a Xorb uploaded through an upload grant and stores it. Until it is committed, the Xorb does not exist, and Shards that reference it are rejected.
- Path:
/v1/xorb-commits/{prefix}/{hash}/grant/{grant_id} - Method:
POST - Parameters:
prefix:default.hash: Xorb hash in hex format (64 lowercase hexadecimal characters).grant_id: theidfrom the grant, URL-encoded.
- Minimum Token Scope:
write - Body: None.
- Response:
200 OKwith an empty body. Also returned when the Xorb already existed (for example, committed by a concurrent upload). - Error Responses: See Error Cases
400 Bad Request:grant_idis malformed, or the uploaded data is not a valid Xorb or does not matchhash. Not retryable.401 Unauthorized: The token is missing, invalid, or expired, or does not havewritescope (for example, areadtoken). Refresh the token, or provide awritescope token.404 Not Found: No uploaded data was found for this grant. Request a new grant and upload again.
POST /v1/xorb-commits/default/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef/grant/01JBX6Q0Z8K4T3V9M2N5P7R1SA -H "Authorization: Bearer <token>"
13. Upload Telemetry
Description: Accepts one client telemetry report. OPTIONAL for clients. See Telemetry for what
hf_xetsends and how to opt out.Path:
/v1/telemetryMethod:
POSTHeaders:
Content-Type:application/json.
Minimum Token Scope:
readBody: JSON, at most 5 KiB. Other top-level fields are ignored.
{ "time": "2026-07-27T10:00:00.000Z", "event": "upload_complete", "session_id": "abc-123", "user_agent": "hf-xet/1.8.0", "metrics": { "bytes": 42 } }time: RFC 3339 timestamp.event,session_id: strings of at most 128 bytes.user_agent: string of at most 512 bytes.metrics: JSON object with at most 128 keys in total, each key at most 128 bytes, nested at most 4 levels deep.
Response:
200 OKwith an empty body.Error Responses: See Error Cases
400 Bad Request: The body is not valid JSON, a field is missing or has the wrong type,timeis not RFC 3339, or a limit above is exceeded.401 Unauthorized: Refresh the token to continue making requests, or provide a token in theAuthorizationheader.413 Payload Too Large: The body is larger than 5 KiB.
POST /v1/telemetry -H "Authorization: Bearer <token>" -H "Content-Type: application/json"
Error Cases
Non-Retryable Errors
- 400 Bad Request: Returned when the request parameters are invalid (for example, invalid Xorb/Shard on upload APIs).
- 401 Unauthorized: The token is missing, invalid, or expired, or does not have the scope the API requires (for example, a
readtoken on an upload API). Refresh the token, or request one with the required scope. - 404 Not Found: Occurs on
GETAPIs where the resource (Xorb, file) does not exist. - 416 Range Not Satisfiable: Reconstruction API only; returned when byte range requests are invalid. Specifically, the requested start range is greater than or equal to the length of the file.
Retryable Errors
- Connection Errors: Often caused by network issues. Retry if intermittent.
Clients SHOULD ensure no firewall blocks requests and SHOULD NOT use DNS overrides.
With upload grants, Xorb data is uploaded to object storage hosts such as
s3.us-east-1.amazonaws.comands3.eu-west-3.amazonaws.comrather than to the CAS server; if those hosts are blocked, clients fall back toPOST /v1/xorbs, which can be slower. - 429 Rate Limiting: Lower your request rate using a backoff strategy, then wait and retry. Assume all APIs are rate limited.
- 500 Internal Server Error: The server experienced an intermittent issue; clients SHOULD retry their requests.
- 503 Service Unavailable: Service is temporarily unable to process requests; wait and retry.
- 504 Gateway Timeout: Service took too long to respond; wait and retry.