diff --git a/src/routes/blog/post/announcing-s3-api/+page.markdoc b/src/routes/blog/post/announcing-s3-api/+page.markdoc new file mode 100644 index 0000000000..e714ce11a2 --- /dev/null +++ b/src/routes/blog/post/announcing-s3-api/+page.markdoc @@ -0,0 +1,156 @@ +--- +layout: post +title: "Announcing the S3 API: Use any S3 client with Appwrite Storage" +description: Appwrite Storage now exposes an S3-compatible API. Point the AWS CLI, the AWS SDKs, and tools like rclone at your Appwrite buckets, with no migration required. +date: 2026-07-31 +cover: /images/blog/announcing-s3-api/cover.avif +timeToRead: 5 +author: aditya-oberai +category: announcement +featured: false +callToAction: true +faqs: + - question: "What is the Appwrite S3 API?" + answer: "It is an S3-compatible API for Appwrite Storage. It authenticates requests with AWS Signature Version 4 and maps standard S3 operations onto Appwrite buckets and files, so existing S3 clients, SDKs, and tools work after you change the endpoint, credentials, and region." + - question: "Which credentials do I use with an S3 client?" + answer: "Your Appwrite project ID is the access key ID, and an Appwrite API key secret is the secret access key. The API key needs the storage scopes that match your workload: `buckets.read` and `files.read` for read-only access, plus `buckets.write` and `files.write` for uploads, deletes, and bucket creation." + - question: "Do I need to migrate my existing buckets and files?" + answer: "No. Buckets and files are addressable over both the native Storage API and the S3 API at the same time. Any bucket you already have in your project is usable over S3 immediately, and objects uploaded over S3 appear in the Appwrite Console like any other file." + - question: "Which S3 operations are supported?" + answer: "The API covers the operations most clients depend on: bucket operations like `ListBuckets`, `CreateBucket`, and `DeleteBucket`; object operations like `PutObject`, `GetObject`, `CopyObject`, `DeleteObjects`, and `ListObjectsV2`; and the full multipart upload flow. Features like object tagging, lifecycle rules, bucket policies, and versioning return `NotImplemented`." + - question: "Does the S3 API support folders?" + answer: "Folders are virtual, as they are on S3 itself: they exist as `/`-separated prefixes in object keys. List a bucket with `delimiter=/` and an optional `prefix` to browse it like a directory tree, with folders returned under `CommonPrefixes` and files under `Contents`." + - question: "How are object keys mapped to Appwrite files?" + answer: "Uploading an object creates an Appwrite file with a generated file ID and your object key as the file name. Read, copy, and delete operations then address the object by its canonical `{fileId}-{fileName}` key, which list operations return. See the [S3 API documentation](/docs/products/storage/s3) for details." +--- + +The S3 API has become the common language of object storage. The AWS CLI, the AWS SDKs, backup tools like rclone and s3cmd, data pipelines, and countless libraries all speak it. Until now, using them with Appwrite Storage meant writing glue code around the native API. + +Today, we are announcing the **S3 API** for Appwrite Storage, an S3-compatible API that lets you point any S3 client, SDK, or tool at your Appwrite buckets and files. + +The API uses AWS Signature Version 4 and maps standard S3 operations onto Appwrite Storage, so most existing S3 code works after you change three settings: the endpoint, the credentials, and the region. + +# Why this matters + +Appwrite Storage already gives you buckets, permissions, image transformations, and file tokens through the native API and SDKs. The S3 API adds a second door into the same storage, unlocking the whole S3 ecosystem: + +- **Bring your existing tools.** Use the AWS CLI, rclone, s3cmd, and any S3-aware library or service against Appwrite buckets without glue code. +- **Reuse existing code.** Applications already written against the AWS SDKs connect to Appwrite by changing the client configuration, not the code. +- **No migration required.** Buckets and files are addressable over both the native Storage API and the S3 API at the same time. Files uploaded over S3 show up in the Appwrite Console, and files uploaded through the Console or SDKs are listable and downloadable over S3. +- **Multipart uploads out of the box.** SDK transfer managers and `aws s3 cp` upload large files in parts automatically, with the standard multipart operations fully supported. + +# How it works + +S3 clients authenticate with an access key ID and a secret access key. Appwrite maps these to your project ID and an [API key](/docs/advanced/security/api-keys) secret: + +| Setting | Value | +| --- | --- | +| Endpoint | `https://.cloud.appwrite.io/v1/s3` | +| Access key ID | Your Appwrite project ID | +| Secret access key | An Appwrite API key secret | +| Region | `auto` | +| Signature version | AWS Signature Version 4 (SigV4) | +| Addressing style | Path-style only | + +The API key's scopes control what the client can do: grant `buckets.read` and `files.read` for read-only workloads, and add `buckets.write` and `files.write` for uploads, deletes, and bucket creation. + +Here is the same configuration in the AWS CLI, the AWS SDK for JavaScript, and boto3: + +{% multicode %} +```bash +# AWS CLI +aws configure set aws_access_key_id +aws configure set aws_secret_access_key +aws configure set region auto + +# Appwrite serves path-style URLs only, so force path addressing +aws configure set default.s3.addressing_style path + +# Pass the Appwrite endpoint on every command +aws s3 ls --endpoint-url https://.cloud.appwrite.io/v1/s3 +``` + +```js +// Node.js: @aws-sdk/client-s3 (v3) +import { S3Client } from '@aws-sdk/client-s3'; + +const client = new S3Client({ + endpoint: 'https://.cloud.appwrite.io/v1/s3', + region: 'auto', + forcePathStyle: true, + credentials: { + accessKeyId: '', + secretAccessKey: '' + } +}); +``` + +```python +# Python: boto3 +import boto3 +from botocore.config import Config + +s3 = boto3.client( + 's3', + endpoint_url='https://.cloud.appwrite.io/v1/s3', + region_name='auto', + aws_access_key_id='', + aws_secret_access_key='', + config=Config(signature_version='s3v4', s3={'addressing_style': 'path'}) +) +``` +{% /multicode %} + +Once the client is configured, everyday S3 commands work as usual: + +```bash +# Create a bucket +aws s3api create-bucket --bucket my-bucket \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 + +# Upload a file (uses multipart automatically for large files) +aws s3 cp ./january.pdf s3://my-bucket/reports/january.pdf \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 + +# List objects to discover canonical keys +aws s3 ls s3://my-bucket --recursive \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 +``` + +# S3 objects, Appwrite files + +S3 buckets map directly to Appwrite Storage buckets, and S3 objects map to files. When you upload an object, Appwrite stores it as a file with a generated file ID and uses your object key as the file's name. Read, copy, and delete operations then address the object by its **canonical key**, which list operations return: + +```text +{fileId}-{fileName} +``` + +This means files created through the native Storage API or the Console are usable over S3 too: list the bucket to discover a file's canonical key, then address it with that key. + +Folders work the way they do on S3 itself: they are virtual, existing as `/`-separated prefixes in object keys. List a bucket with the `/` delimiter and an optional `prefix` to browse it like a directory tree, with folders grouped under `CommonPrefixes` and files under `Contents`. + +# What's supported + +The S3 API targets the operations most clients depend on: + +- **Bucket operations**: `ListBuckets`, `CreateBucket`, `HeadBucket`, `DeleteBucket`, and `GetBucketLocation` +- **Object operations**: `PutObject`, `GetObject` (with `Range` and conditional requests), `HeadObject`, `CopyObject`, `DeleteObject`, `DeleteObjects`, `ListObjects`, and `ListObjectsV2` +- **Multipart uploads**: the full flow, from `CreateMultipartUpload` through `UploadPart`, `UploadPartCopy`, `CompleteMultipartUpload`, `AbortMultipartUpload`, `ListMultipartUploads`, and `ListParts` + +Buckets keep their constraints under the S3 API: uploads that exceed the bucket's maximum file size or use a disallowed extension are rejected, and access is governed by [Appwrite permissions](/docs/products/storage/permissions) and API key scopes. Features outside this set, such as object tagging, lifecycle rules, bucket policies, and versioning, return standard `NotImplemented` errors, so clients fail cleanly. The full list is in the [limitations documentation](/docs/products/storage/s3#limitations). + +# Get started + +The S3 API is available on **Appwrite Cloud** today. + +1. Navigate to **Overview** > **Integrations** > **API keys** and create an API key with the storage scopes your workload needs. +2. Configure your S3 client with the `https://.cloud.appwrite.io/v1/s3` endpoint, your project ID as the access key ID, and the API key secret as the secret access key. +3. Run your existing S3 commands and code against your Appwrite buckets. + +Full documentation, including object key mapping, folder listings, and error responses, is available on the [S3 API documentation page](/docs/products/storage/s3). + +# Resources + +- [S3 API documentation](/docs/products/storage/s3) +- [Storage documentation](/docs/products/storage) +- [API keys documentation](/docs/advanced/security/api-keys) diff --git a/src/routes/changelog/(entries)/2026-07-31.markdoc b/src/routes/changelog/(entries)/2026-07-31.markdoc new file mode 100644 index 0000000000..8fbc7a863e --- /dev/null +++ b/src/routes/changelog/(entries)/2026-07-31.markdoc @@ -0,0 +1,16 @@ +--- +layout: changelog +title: "Announcing the S3 API for Appwrite Storage" +date: 2026-07-31 +cover: /images/blog/announcing-s3-api/cover.avif +--- + +Appwrite Storage now exposes an **S3-compatible API**. Point the AWS CLI, the AWS SDKs, and tools like rclone or s3cmd at your Appwrite buckets by changing three settings: the endpoint (`https://.cloud.appwrite.io/v1/s3`), the credentials (your project ID and an API key secret), and the region (`auto`). + +The API uses AWS Signature Version 4 and covers the operations most clients depend on, including bucket management, object uploads and downloads, copy and batch delete, folder-style listings with `delimiter=/`, and the full multipart upload flow. It works alongside your existing data: buckets and files are addressable over both the native Storage API and the S3 API at the same time, with no migration required. + +The S3 API is available on Appwrite Cloud today. + +{% arrow_link href="/blog/post/announcing-s3-api" %} +Read the announcement +{% /arrow_link %} diff --git a/src/routes/docs/products/storage/+layout.svelte b/src/routes/docs/products/storage/+layout.svelte index 122404835c..8ac86d359d 100644 --- a/src/routes/docs/products/storage/+layout.svelte +++ b/src/routes/docs/products/storage/+layout.svelte @@ -28,6 +28,10 @@ label: 'Buckets', href: '/docs/products/storage/buckets' }, + { + label: 'Folders', + href: '/docs/products/storage/folders' + }, { label: 'Permissions', href: '/docs/products/storage/permissions' @@ -48,6 +52,10 @@ { label: 'Image transformations', href: '/docs/products/storage/images' + }, + { + label: 'S3 API', + href: '/docs/products/storage/s3' } ] }, diff --git a/src/routes/docs/products/storage/buckets/+page.markdoc b/src/routes/docs/products/storage/buckets/+page.markdoc index fb5f76e494..004f9b2691 100644 --- a/src/routes/docs/products/storage/buckets/+page.markdoc +++ b/src/routes/docs/products/storage/buckets/+page.markdoc @@ -296,6 +296,13 @@ This means users can't create new files or read, update, and delete existing fil [Learn about configuring permissions](/docs/products/storage/permissions). +# Folders {% #folders %} +Files in a bucket can be organized into virtual folders. +Each file carries a folder path, like `photos/2026`, that you set when uploading it. +Folders are derived from these paths, so there is nothing to create or manage separately: a folder appears when the first file is uploaded into it and disappears when the last file inside it is deleted. + +[Learn about working with folders](/docs/products/storage/folders). + # Encryption {% #encryption %} Appwrite provides added security settings for your buckets. Enable encryption under your bucket's **Settings** > **Security settings**. You can enable encryption to encrypt files in your buckets. If your files are leaked, encrypted files cannot be read by the malicious actor. diff --git a/src/routes/docs/products/storage/folders/+page.markdoc b/src/routes/docs/products/storage/folders/+page.markdoc new file mode 100644 index 0000000000..4da4c4ed92 --- /dev/null +++ b/src/routes/docs/products/storage/folders/+page.markdoc @@ -0,0 +1,639 @@ +--- +layout: article +title: Folders +description: Organize files in Appwrite Storage buckets with virtual folders. Learn how to upload files into folders, list files by folder, and browse folders. +--- + +Appwrite Storage lets you organize the files inside a bucket using virtual folders. +Folders work like key prefixes in S3-compatible storage services: they are derived from the paths of your files, so you never create or delete folders explicitly. + +# How folders work {% #how-folders-work %} + +A bucket doesn't store folders as records. Instead, every file has a `folder` attribute, a path like `photos/2026/`, and folders are derived from these paths: a folder exists whenever at least one file's `folder` path places the file inside it. +For example, uploading a single file with the folder `photos/2026` is what brings both the `photos/` and `photos/2026/` folders into existence. + +A file's folder is set once, when the file is uploaded, and defaults to the bucket root (an empty string). +Folder paths are stored in a canonical form that always ends with a trailing slash, like `photos/2026/`. +Each file also exposes a computed `key` attribute, which is the file's full virtual path: the folder followed by the file name, like `photos/2026/Pink.png`. + +Because folders are derived from files, they exist implicitly. A folder appears as soon as the first file is uploaded into it and disappears when the last file inside it is deleted. +There are no empty folders, no folder permissions, and no folder metadata to manage. + +Unlike in S3, uploading a file with the same name to the same folder does not overwrite the existing file. +Files are identified by their file ID, so multiple files can share the same `key`. + +# Upload files to a folder {% #upload-files-to-a-folder %} + +To place a file inside a folder, pass the optional `folder` parameter when [uploading the file](/docs/products/storage/upload-download#create-file). +Nest folders using `/`, for example `photos/2026`. The trailing slash is optional on input and is added automatically when stored. + +{% multicode %} + ```client-web + import { Client, Storage, ID } from "appwrite"; + + const client = new Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + + const storage = new Storage(client); + + const file = await storage.createFile({ + bucketId: '', + fileId: ID.unique(), + file: document.getElementById('uploader').files[0], + folder: 'photos/2026' + }); + + console.log(file.folder); // 'photos/2026/' + console.log(file.key); // 'photos/2026/Pink.png' + ``` + + ```server-nodejs + const sdk = require('node-appwrite'); + const { InputFile } = require('node-appwrite/file'); + + const client = new sdk.Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject('') + .setKey(''); + + const storage = new sdk.Storage(client); + + const file = await storage.createFile({ + bucketId: '', + fileId: sdk.ID.unique(), + file: InputFile.fromPath('/path/to/Pink.png', 'Pink.png'), + folder: 'photos/2026' + }); + + console.log(file.folder); // 'photos/2026/' + console.log(file.key); // 'photos/2026/Pink.png' + ``` + + ```client-flutter + import 'package:appwrite/appwrite.dart'; + + void main() async { + final client = Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + + final storage = Storage(client); + + final file = await storage.createFile( + bucketId: '', + fileId: ID.unique(), + file: InputFile.fromPath(path: './path-to-files/Pink.png', filename: 'Pink.png'), + folder: 'photos/2026', + ); + } + ``` + + ```client-android-kotlin + import io.appwrite.Client + import io.appwrite.ID + import io.appwrite.models.InputFile + import io.appwrite.services.Storage + + suspend fun main() { + val client = Client(applicationContext) + .setEndpoint("https://.cloud.appwrite.io/v1") // Your API Endpoint + .setProject("") // Your project ID + + val storage = Storage(client) + + val file = storage.createFile( + bucketId = "", + fileId = ID.unique(), + file = InputFile.fromPath("./path-to-files/Pink.png"), + folder = "photos/2026", + ) + } + ``` + + ```client-apple + import Appwrite + + func main() async throws { + let client = Client() + .setEndpoint("https://.cloud.appwrite.io/v1") + .setProject("") + + let storage = Storage(client) + + let file = try await storage.createFile( + bucketId: "", + fileId: ID.unique(), + file: InputFile.fromBuffer(yourByteBuffer, + filename: "Pink.png", + mimeType: "image/png" + ), + folder: "photos/2026" + ) + } + ``` + + ```client-react-native + import { Client, Storage, ID } from 'react-native-appwrite'; + + const client = new Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + + const storage = new Storage(client); + + const file = await storage.createFile({ + bucketId: '', + fileId: ID.unique(), + file: { + name: 'Pink.png', + type: 'image/png', + size: 1234567, + uri: 'file:///path/to/Pink.png', + }, + folder: 'photos/2026' + }); + ``` +{% /multicode %} + +{% info title="Folder naming rules" %} +Folder paths are `/`-separated segments. A folder path must not start with `/`, must not contain empty, `.`, or `..` segments or control characters, and can be at most 2,048 characters long including the trailing slash. +{% /info %} + +A file's folder can't be changed after upload. To move a file into a different folder, create the file again with the new folder and delete the original. + +# List files in a folder {% #list-files-in-a-folder %} + +Filter files by folder using [queries](/docs/products/databases/queries) on the `folder` attribute when listing files. + +| Goal | Query | +| ---- | ----- | +| Files directly inside `photos/2026/` | `Query.equal('folder', ['photos/2026/'])` | +| Files at the bucket root only | `Query.equal('folder', [''])` | +| Files anywhere under `photos/`, including nested folders | `Query.startsWith('folder', 'photos/')` | + +Query values must match the stored form of the folder path, which always includes the trailing slash. +For example, `Query.equal('folder', ['photos/2026/'])` matches files in `photos/2026/`, but `Query.equal('folder', ['photos/2026'])` matches nothing. + +{% multicode %} + ```client-web + import { Client, Storage, Query } from "appwrite"; + + const client = new Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + + const storage = new Storage(client); + + const result = await storage.listFiles({ + bucketId: '', + queries: [ + Query.equal('folder', ['photos/2026/']) + ] + }); + + console.log(result.files); + ``` + + ```server-nodejs + const sdk = require('node-appwrite'); + + const client = new sdk.Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject('') + .setKey(''); + + const storage = new sdk.Storage(client); + + const result = await storage.listFiles({ + bucketId: '', + queries: [ + sdk.Query.equal('folder', ['photos/2026/']) + ] + }); + + console.log(result.files); + ``` + + ```client-flutter + import 'package:appwrite/appwrite.dart'; + + void main() async { + final client = Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + + final storage = Storage(client); + + final result = await storage.listFiles( + bucketId: '', + queries: [ + Query.equal('folder', ['photos/2026/']) + ], + ); + } + ``` + + ```client-android-kotlin + import io.appwrite.Client + import io.appwrite.Query + import io.appwrite.services.Storage + + suspend fun main() { + val client = Client(applicationContext) + .setEndpoint("https://.cloud.appwrite.io/v1") // Your API Endpoint + .setProject("") // Your project ID + + val storage = Storage(client) + + val result = storage.listFiles( + bucketId = "", + queries = listOf( + Query.equal("folder", listOf("photos/2026/")) + ) + ) + } + ``` + + ```client-apple + import Appwrite + + func main() async throws { + let client = Client() + .setEndpoint("https://.cloud.appwrite.io/v1") + .setProject("") + + let storage = Storage(client) + + let result = try await storage.listFiles( + bucketId: "", + queries: [ + Query.equal("folder", value: ["photos/2026/"]) + ] + ) + } + ``` + + ```client-react-native + import { Client, Storage, Query } from 'react-native-appwrite'; + + const client = new Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + + const storage = new Storage(client); + + const result = await storage.listFiles({ + bucketId: '', + queries: [ + Query.equal('folder', ['photos/2026/']) + ] + }); + ``` +{% /multicode %} + +# List folders {% #list-folders %} + +To browse the folders inside a bucket, use the `listFolders` method. +It returns the folders directly inside a given folder, or the top-level folders of the bucket when no folder is passed. + +{% multicode %} + ```client-web + import { Client, Storage } from "appwrite"; + + const client = new Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + + const storage = new Storage(client); + + const result = await storage.listFolders({ + bucketId: '', + folder: 'photos/' // Omit to list top-level folders + }); + + console.log(result.folders); + ``` + + ```server-nodejs + const sdk = require('node-appwrite'); + + const client = new sdk.Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject('') + .setKey(''); + + const storage = new sdk.Storage(client); + + const result = await storage.listFolders({ + bucketId: '', + folder: 'photos/' // Omit to list top-level folders + }); + + console.log(result.folders); + ``` + + ```client-flutter + import 'package:appwrite/appwrite.dart'; + + void main() async { + final client = Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + + final storage = Storage(client); + + final result = await storage.listFolders( + bucketId: '', + folder: 'photos/', // Omit to list top-level folders + ); + } + ``` + + ```client-android-kotlin + import io.appwrite.Client + import io.appwrite.services.Storage + + suspend fun main() { + val client = Client(applicationContext) + .setEndpoint("https://.cloud.appwrite.io/v1") // Your API Endpoint + .setProject("") // Your project ID + + val storage = Storage(client) + + val result = storage.listFolders( + bucketId = "", + folder = "photos/" // Omit to list top-level folders + ) + } + ``` + + ```client-apple + import Appwrite + + func main() async throws { + let client = Client() + .setEndpoint("https://.cloud.appwrite.io/v1") + .setProject("") + + let storage = Storage(client) + + let result = try await storage.listFolders( + bucketId: "", + folder: "photos/" // Omit to list top-level folders + ) + } + ``` + + ```client-react-native + import { Client, Storage } from 'react-native-appwrite'; + + const client = new Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + + const storage = new Storage(client); + + const result = await storage.listFolders({ + bucketId: '', + folder: 'photos/' // Omit to list top-level folders + }); + ``` +{% /multicode %} + +Each folder in the response has a `key` (the full folder path), a `name` (the last segment of the path), and a `parent` (the path of the folder that contains it). + +```json +{ + "folders": [ + { + "key": "photos/2025/", + "name": "2025", + "parent": "photos/" + }, + { + "key": "photos/2026/", + "name": "2026", + "parent": "photos/" + } + ] +} +``` + +## Pagination {% #pagination %} + +`listFolders` returns up to `limit` folders per page, 25 by default and 100 at most, sorted by path. +Because folders are derived from files instead of being stored as records, the response doesn't include a total count. + +To fetch the next page, pass the `key` of the last folder you received as the `cursor` parameter. +When a page comes back with fewer folders than the limit, you've reached the end of the listing. + +{% multicode %} + ```client-web + import { Client, Storage } from "appwrite"; + + const client = new Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + + const storage = new Storage(client); + + const limit = 25; + let cursor = ''; + let folders = []; + + while (true) { + const page = await storage.listFolders({ + bucketId: '', + folder: 'photos/', + limit, + cursor + }); + + folders.push(...page.folders); + + if (page.folders.length < limit) { + break; // Listing is complete + } + + cursor = page.folders[page.folders.length - 1].key; + } + ``` + + ```server-nodejs + const sdk = require('node-appwrite'); + + const client = new sdk.Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject('') + .setKey(''); + + const storage = new sdk.Storage(client); + + const limit = 25; + let cursor = ''; + let folders = []; + + while (true) { + const page = await storage.listFolders({ + bucketId: '', + folder: 'photos/', + limit, + cursor + }); + + folders.push(...page.folders); + + if (page.folders.length < limit) { + break; // Listing is complete + } + + cursor = page.folders[page.folders.length - 1].key; + } + ``` + + ```client-flutter + import 'package:appwrite/appwrite.dart'; + import 'package:appwrite/models.dart'; + + void main() async { + final client = Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + + final storage = Storage(client); + + const limit = 25; + var cursor = ''; + final folders = []; + + while (true) { + final page = await storage.listFolders( + bucketId: '', + folder: 'photos/', + limit: limit, + cursor: cursor, + ); + + folders.addAll(page.folders); + + if (page.folders.length < limit) { + break; // Listing is complete + } + + cursor = page.folders.last.key; + } + } + ``` + + ```client-android-kotlin + import io.appwrite.Client + import io.appwrite.models.Folder + import io.appwrite.services.Storage + + suspend fun main() { + val client = Client(applicationContext) + .setEndpoint("https://.cloud.appwrite.io/v1") // Your API Endpoint + .setProject("") // Your project ID + + val storage = Storage(client) + + val limit = 25L + var cursor = "" + val folders = mutableListOf() + + while (true) { + val page = storage.listFolders( + bucketId = "", + folder = "photos/", + limit = limit, + cursor = cursor + ) + + folders.addAll(page.folders) + + if (page.folders.size < limit) { + break // Listing is complete + } + + cursor = page.folders.last().key + } + } + ``` + + ```client-apple + import Appwrite + import AppwriteModels + + func main() async throws { + let client = Client() + .setEndpoint("https://.cloud.appwrite.io/v1") + .setProject("") + + let storage = Storage(client) + + let limit = 25 + var cursor = "" + var folders: [AppwriteModels.Folder] = [] + + while true { + let page = try await storage.listFolders( + bucketId: "", + folder: "photos/", + limit: limit, + cursor: cursor + ) + + folders.append(contentsOf: page.folders) + + if page.folders.count < limit { + break // Listing is complete + } + + cursor = page.folders.last!.key + } + } + ``` + + ```client-react-native + import { Client, Storage } from 'react-native-appwrite'; + + const client = new Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + + const storage = new Storage(client); + + const limit = 25; + let cursor = ''; + let folders = []; + + while (true) { + const page = await storage.listFolders({ + bucketId: '', + folder: 'photos/', + limit, + cursor + }); + + folders.push(...page.folders); + + if (page.folders.length < limit) { + break; // Listing is complete + } + + cursor = page.folders[page.folders.length - 1].key; + } + ``` +{% /multicode %} + +The cursor must be a folder directly inside the folder you are listing. Passing any other path returns an error. + +# Permissions {% #permissions %} + +Folders don't carry their own permissions. Access is derived from the files inside them. +When a bucket uses [file security](/docs/products/storage/permissions), listing folders only returns folders that contain at least one file the user can read. + +{% arrow_link href="/docs/products/storage/permissions" %} +Learn more about storage permissions +{% /arrow_link %} diff --git a/src/routes/docs/products/storage/s3/+page.markdoc b/src/routes/docs/products/storage/s3/+page.markdoc new file mode 100644 index 0000000000..4a61b38b45 --- /dev/null +++ b/src/routes/docs/products/storage/s3/+page.markdoc @@ -0,0 +1,287 @@ +--- +layout: article +title: S3 API +description: Connect any S3-compatible client, SDK, or tool to Appwrite Storage. Configure credentials once, then manage buckets, objects, and multipart uploads over the S3 API. +--- + +Appwrite Storage exposes an S3-compatible API, so you can point the AWS CLI, the AWS SDKs, and third-party tools like rclone or s3cmd at your buckets and files. The API uses AWS Signature Version 4 and maps standard S3 operations onto Appwrite Storage, so most existing S3 code works after you change three settings: the endpoint, the credentials, and the region. + +The S3 API is available on Appwrite Cloud. It needs a project ID and an [API key](/docs/advanced/security/api-keys) with the [storage scopes](#api-key-scopes) described below. It also works alongside your existing data: buckets and files are addressable over both the native Storage API and the S3 API at the same time, with no migration required. + +# Connect an S3 client {% #connect-an-s3-client %} + +Configure your client with the following values. S3 clients authenticate with an access key ID and a secret access key, which Appwrite maps to your project ID and an API key secret. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://.cloud.appwrite.io/v1/s3` | +| Access key ID | Your Appwrite project ID | +| Secret access key | An Appwrite API key secret | +| Region | `auto` | +| Signature version | AWS Signature Version 4 (SigV4) | +| Addressing style | Path-style only | + +The `` in the endpoint is your Appwrite Cloud region (for example, `fra` or `nyc`), the same region you use for the rest of the Appwrite API. The S3 `region` setting is separate: some clients require a region to sign requests, but Appwrite ignores the value, so set it to `auto`. + +The examples below apply these settings in the AWS CLI and the AWS SDKs. + +{% multicode %} +```bash +# AWS CLI +aws configure set aws_access_key_id +aws configure set aws_secret_access_key +aws configure set region auto + +# Appwrite serves path-style URLs only, so force path addressing +aws configure set default.s3.addressing_style path + +# Pass the Appwrite endpoint on every command +aws s3 ls --endpoint-url https://.cloud.appwrite.io/v1/s3 +``` + +```js +// Node.js: @aws-sdk/client-s3 (v3) +import { S3Client } from '@aws-sdk/client-s3'; + +const client = new S3Client({ + endpoint: 'https://.cloud.appwrite.io/v1/s3', + region: 'auto', + forcePathStyle: true, + credentials: { + accessKeyId: '', + secretAccessKey: '' + } +}); +``` + +```python +# Python: boto3 +import boto3 +from botocore.config import Config + +s3 = boto3.client( + 's3', + endpoint_url='https://.cloud.appwrite.io/v1/s3', + region_name='auto', + aws_access_key_id='', + aws_secret_access_key='', + config=Config(signature_version='s3v4', s3={'addressing_style': 'path'}) +) +``` + +```go +// Go: aws-sdk-go-v2 +package main + +import ( + "context" + "log" + + "github.com/aws/aws-sdk-go-v2/aws" + "github.com/aws/aws-sdk-go-v2/config" + "github.com/aws/aws-sdk-go-v2/credentials" + "github.com/aws/aws-sdk-go-v2/service/s3" +) + +func main() { + cfg, err := config.LoadDefaultConfig(context.TODO(), + config.WithRegion("auto"), + config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider( + "", "", "", + )), + ) + if err != nil { + log.Fatal(err) + } + + client := s3.NewFromConfig(cfg, func(o *s3.Options) { + o.BaseEndpoint = aws.String("https://.cloud.appwrite.io/v1/s3") + o.UsePathStyle = true + }) + + buckets, err := client.ListBuckets(context.TODO(), &s3.ListBucketsInput{}) + if err != nil { + log.Fatal(err) + } + + for _, bucket := range buckets.Buckets { + log.Println(*bucket.Name) + } +} +``` + +```php +// PHP: aws/aws-sdk-php +use Aws\S3\S3Client; + +$client = new S3Client([ + 'version' => 'latest', + 'region' => 'auto', + 'endpoint' => 'https://.cloud.appwrite.io/v1/s3', + 'use_path_style_endpoint' => true, + 'credentials' => [ + 'key' => '', + 'secret' => '', + ], +]); +``` +{% /multicode %} + +# API key scopes {% #api-key-scopes %} + +Create an API key in the Appwrite Console under **Overview** > **Integrations** > **API keys**, then grant the storage scopes your workload needs. The project ID identifies your project, while the API key secret both authenticates the request and determines what it is allowed to do. + +| Access level | Required scopes | +| --- | --- | +| Read-only (list and download) | `buckets.read`, `files.read` | +| Write (create buckets, upload, delete) | `buckets.write`, `files.write` | +| Copy (`CopyObject`, `UploadPartCopy`) | `files.read`, `files.write` | +| Full read and write | `buckets.read`, `files.read`, `buckets.write`, `files.write` | + +Copying an object reads the source and writes the destination, so `CopyObject` and `UploadPartCopy` need both `files.read` and `files.write`. A request signed with a secret that does not match a project API key, or a key missing the required scope, is rejected with `AccessDenied` (HTTP 403). + +# Object key mapping {% #object-key-mapping %} + +S3 buckets map directly to Appwrite Storage buckets, and S3 objects map to files. Creating a bucket over S3 creates an Appwrite bucket that uses the S3 bucket name as both its ID and its name, with Appwrite's default settings. Any bucket you already have in your project is usable over S3 immediately. + +When you upload an object, Appwrite stores it as a file with an automatically generated file ID and uses the object key you provided as the file's name. Read, copy, and delete operations then address the object by its **canonical key**, which list operations return: + +```text +{fileId}-{fileName} +``` + +For example, uploading to the key `reports/january.pdf` creates a file whose name is `reports/january.pdf`, and the object is then listed under a canonical key such as `6710e2f300048d1e2c3f-reports/january.pdf`. + +The same flow covers files created through the native Storage API or the Console: list the bucket to discover a file's canonical key, then address it over S3 with that key. + +{% info title="Uploads never overwrite by key" %} +Because a new file ID is generated on every plain upload, uploading twice with the same key creates two distinct objects instead of overwriting. To read, copy, overwrite, or delete an object, use its canonical key from a list operation. Uploading to a canonical key that already exists overwrites that object in place. +{% /info %} + +# Folders {% #folders %} + +Appwrite Storage buckets are flat. There is no folder resource, and folders are never created or deleted on their own: they exist only as `/`-separated prefixes in object keys. Uploading to `reports/2024/january.pdf` stores a single object whose name contains those prefixes; it does not create `reports/` or `reports/2024/` as separate entities. + +To browse a bucket like a directory tree, list objects with the `/` delimiter and, optionally, a `prefix`. Appwrite groups object names at each `/` boundary: names sharing a prefix up to the next `/` collapse into a single entry under `CommonPrefixes` (the "folders"), while objects at the current level are returned under `Contents` (the "files"). + +Grouping is applied to the object's name (the key you uploaded), so folder prefixes come back clean, without the generated file ID. Pass a `prefix` such as `reports/` to open that folder and list one level deeper. Listing without a delimiter returns every matching object with its full name, which is how you enumerate a bucket recursively. + +Objects under `Contents` are still identified by their canonical `{fileId}-{fileName}` key (see [Object key mapping](#object-key-mapping)); use that key to read, copy, or delete a file. `CommonPrefixes` are folder paths only, with nothing to download at the prefix itself. + +`/` is the only supported delimiter. Setting `delimiter` to any other value returns `NotImplemented` (HTTP 501). + +```bash +# List the top level of a bucket: immediate folders (CommonPrefixes) and files +aws s3api list-objects-v2 --bucket my-bucket --delimiter / \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 + +# Open the "reports" folder and list one level down +aws s3api list-objects-v2 --bucket my-bucket --prefix reports/ --delimiter / \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 + +# aws s3 ls uses the / delimiter automatically; omit --recursive to browse by folder +aws s3 ls s3://my-bucket/reports/ \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 +``` + +# Example commands {% #example-commands %} + +Once your client is configured, everyday S3 commands work as usual. These AWS CLI examples assume you have run `aws configure` as shown above; each command passes the endpoint explicitly with `--endpoint-url`. + +```bash +# Create a bucket +aws s3api create-bucket --bucket my-bucket \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 + +# Upload a file (uses multipart automatically for large files) +aws s3 cp ./january.pdf s3://my-bucket/reports/january.pdf \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 + +# List objects to discover canonical keys +aws s3 ls s3://my-bucket --recursive \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 + +# Download using the canonical {fileId}-{fileName} key from the listing +aws s3api get-object --bucket my-bucket \ + --key 6710e2f300048d1e2c3f-reports/january.pdf ./january.pdf \ + --endpoint-url https://.cloud.appwrite.io/v1/s3 +``` + +# Bucket operations {% #bucket-operations %} + +All operations are addressed with path-style URLs under `/v1/s3`. The following bucket operations are supported. + +| Operation | Description | +| --- | --- | +| `ListBuckets` | List the project's buckets. | +| `CreateBucket` | Create a bucket. The S3 bucket name becomes the Appwrite bucket's ID and name, with Appwrite's default settings. | +| `HeadBucket` | Check that a bucket exists and is accessible. | +| `DeleteBucket` | Delete an empty bucket. Deleting a bucket that still contains objects returns `BucketNotEmpty` (HTTP 409). | +| `GetBucketLocation` | Returns `us-east-1`. | +| `GetBucketAcl`, `PutBucketAcl` | Compatibility responses only. See [Limitations](#limitations). | + +# Object operations {% #object-operations %} + +The following object operations are supported. + +| Operation | Description | +| --- | --- | +| `PutObject` | Upload an object. Returns an `ETag`. Honors `Content-Type`, `x-amz-meta-*` user metadata, and `x-amz-server-side-encryption`. | +| `GetObject` | Download an object. Supports `Range` requests (HTTP 206), and the `If-None-Match` (HTTP 304) and `If-Match` (HTTP 412) conditional headers. | +| `HeadObject` | Retrieve an object's metadata (size, content type, `ETag`, user metadata) without the body. | +| `CopyObject` | Copy an object using the `x-amz-copy-source` header. Supports `x-amz-metadata-directive: REPLACE` to replace metadata. | +| `DeleteObject` | Delete a single object. | +| `DeleteObjects` | Delete multiple objects in one request, including quiet mode. | +| `ListObjects` | List objects in a bucket, with `prefix` filtering. | +| `ListObjectsV2` | List objects with `prefix`, `delimiter` (only `/`), `max-keys`, `continuation-token`, and `start-after`. Use `delimiter=/` to browse [folders](#folders). | +| `GetObjectAcl`, `PutObjectAcl` | Compatibility responses only. See [Limitations](#limitations). | + +Content type is taken from the `Content-Type` header you send. When that is missing or generic (`application/octet-stream`), Appwrite infers a type from the file extension. Buckets keep their constraints under the S3 API: uploads that exceed the bucket's maximum file size or use a disallowed extension are rejected, and disabled buckets are treated as not found. + +# Multipart uploads {% #multipart-uploads %} + +Large files are uploaded in parts. Standard SDK multipart helpers, such as the AWS CLI's `aws s3 cp` and the SDK transfer managers, use these operations automatically. + +| Operation | Description | +| --- | --- | +| `CreateMultipartUpload` | Begin a multipart upload and receive an `UploadId`. | +| `UploadPart` | Upload a single part. Returns the part's `ETag`. | +| `UploadPartCopy` | Upload a part by copying a byte range from another object, using `x-amz-copy-source` and `x-amz-copy-source-range`. | +| `CompleteMultipartUpload` | Assemble the uploaded parts into the final object. | +| `AbortMultipartUpload` | Discard an in-progress multipart upload and its parts. | +| `ListMultipartUploads` | List in-progress multipart uploads in a bucket. | +| `ListParts` | List the parts already uploaded for an `UploadId`. | + +{% info title="Multipart requirements" %} +Parts must be contiguous and start at part number 1. Completing an upload with non-contiguous part numbers (for example, parts 1, 5, and 100) is rejected. SDK multipart helpers upload parts sequentially, so they are unaffected. In-progress uploads are kept for 24 hours before they are cleaned up, so complete or abort within that window. +{% /info %} + +# Limitations {% #limitations %} + +The S3 API targets the operations most clients depend on. Keep the following in mind. + +- **Path-style addressing only.** Virtual-hosted-style URLs (`https://.host/...`) are not supported. Enable path-style addressing in your client, as shown in [Connect an S3 client](#connect-an-s3-client). +- **Signature Version 4 only.** Requests are authenticated with SigV4, and the request timestamp must be within 15 minutes of the server's clock. +- **Objects are addressed by canonical key.** Reads, copies, overwrites, and deletes address objects by their canonical `{fileId}-{fileName}` key from a list operation, not by the raw key you uploaded with. See [Object key mapping](#object-key-mapping). +- **ACLs are compatibility responses.** `GetObjectAcl`, `PutObjectAcl`, `GetBucketAcl`, and `PutBucketAcl` return a canned private ACL and are accepted for tooling compatibility. They do not change access. Manage access with [Appwrite permissions](/docs/products/storage/permissions) and API key scopes instead. +- **No folder resource or folder markers.** Folders are virtual: they are `/`-separated prefixes in object keys, browsed by listing with `delimiter=/`. See [Folders](#folders). Using any other delimiter value or uploading a folder-marker object (a key ending in `/`) returns `NotImplemented` (HTTP 501). +- **Large listings increase latency.** `ListObjects` returns every matching object in a single response, while `ListObjectsV2` honors `max-keys` (capped at 1000) and paginates with `IsTruncated` and a `NextContinuationToken`. No objects are silently dropped. For large buckets, prefer `ListObjectsV2` with pagination. +- **Custom file IDs with hyphens.** The canonical key is split on the first `-`. Objects created through the S3 API use generated IDs without hyphens, so this only affects interop with pre-existing files whose custom ID contains a `-`. A mismatch returns `NoSuchKey` rather than the wrong object. +- **Unsupported S3 features.** Object tagging, CORS, lifecycle, bucket policies and policy status, encryption configuration, ownership controls, notifications, and versioning are not supported. Requests to these sub-resources return `NotImplemented` (HTTP 501). + +# Error responses {% #error-responses %} + +Errors are returned as standard S3 XML error documents with an S3 error code and HTTP status. + +| S3 error code | HTTP status | Meaning | +| --- | --- | --- | +| `NoSuchBucket` | 404 | The bucket does not exist, is disabled, or is not accessible. | +| `NoSuchKey` | 404 | The object key does not exist. | +| `BucketNotEmpty` | 409 | The bucket still contains objects and cannot be deleted. | +| `BucketAlreadyOwnedByYou` | 409 | A bucket with that ID already exists in your project. | +| `AccessDenied` | 403 | The signature is invalid, or the API key is expired or missing a required scope. | +| `InvalidRange` | 416 | The requested `Range` cannot be satisfied. | +| `PreconditionFailed` | 412 | An `If-Match` precondition did not hold. | +| `NotImplemented` | 501 | The requested S3 feature is not supported. | +| `InvalidRequest` | 400 | The request was malformed or violated a bucket constraint. | +| `InternalError` | 500 | An unexpected server error occurred. | diff --git a/src/routes/docs/products/storage/upload-download/+page.markdoc b/src/routes/docs/products/storage/upload-download/+page.markdoc index 518f3450fb..6a49d1c9a5 100644 --- a/src/routes/docs/products/storage/upload-download/+page.markdoc +++ b/src/routes/docs/products/storage/upload-download/+page.markdoc @@ -203,6 +203,12 @@ You can also upload files programmatically using our SDKs: {% /multicode %} +When uploading, you can pass the optional `folder` parameter to organize files into virtual folders, for example `photos/2026`. + +{% arrow_link href="/docs/products/storage/folders" %} +Learn more about folders +{% /arrow_link %} + # Large files {% #large-files %} When you are trying to upload any files above 5MB, you will need to upload them in chunks for better reliability and performance. If you're using an Appwrite SDK, this is handled automatically.