Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ localMetadata/*
.tox
coverage
.DS_Store
CLAUDE.md
165 changes: 165 additions & 0 deletions CLAUDE.md
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want to add this file to .dockerignore to avoid having it in containers.

Maybe we could have it ignored in CI test workflows as well

Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with
code in this repository.

## Project Overview

CloudServer is Scality's open-source, AWS S3-compatible object storage server.
It provides a single S3 API interface to access multiple storage backends
(Scality RING, file, memory, AWS S3, Azure, GCP, etc.). Part of the
S3C/Federation and Zenko stacks.

- **Node.js >= 22** required
- **Package Manager**: Yarn with `--frozen-lockfile`

## Common Commands

### Development

```bash
yarn install --frozen-lockfile # Install dependencies
yarn start # Start S3 + metadata + data servers
yarn dev # Development mode with auto-restart (nodemon)
yarn mem_backend # Start with in-memory backend (S3BACKEND=mem)
yarn start_mongo # Start with MongoDB metadata backend
```

### Testing

```bash
# Unit tests
yarn test # Run unit tests (S3BACKEND=mem)
yarn cover test # Run unit tests with coverage

# Functional tests (require running server)
yarn ft_awssdk # AWS SDK functional tests
yarn ft_awssdk_buckets # Bucket-specific AWS SDK tests
yarn ft_awssdk_versioning # Versioning tests
yarn ft_s3cmd # S3cmd CLI tests
yarn ft_healthchecks # Health check tests
yarn ft_test # Run all functional tests

# Multi-backend tests
yarn multiple_backend_test # Multi-backend unit tests
yarn ft_awssdk_external_backends # External backend functional tests
```

### Linting

```bash
yarn lint # ESLint on all JS files
yarn lint_md # Markdown linting
```

## Architecture

### Entry Points

- `index.js` → Main S3 API server (port 8000)
- `mdserver.js` → Metadata server (port 9990)
- `dataserver.js` → Data storage server (port 9991)
- `pfsserver.js` → Passthrough filesystem server (port 9992)
- `managementAgent.js` → WebSocket management agent (port 8010)

### Core Components

```
lib/
├── server.js # S3Server class - main HTTP server with cluster support
├── Config.js # Configuration management (env vars, config.json)
├── api/ # S3 API implementations (80+ operations)
│ ├── api.js # Main request router and auth handler
│ └── apiUtils/ # Shared utilities (auth, bucket ops, object ops, quotas)
├── auth/ # Authentication (vault.js for IAM, in_memory/ for local)
├── data/ # Data backend abstraction (file, memory, multiple)
├── metadata/ # Metadata backend abstraction
├── routes/ # Special routes (Backbeat, metadata service, Veeam)
├── kms/ # Key management (file, memory, AWS, KMIP)
├── management/ # Management API and configuration
└── utilities/ # Logging, health checks, XML parsing
```

### Backend Configuration

Set via environment variables. Each backend type abstracts multiple
implementations:

### Data Backends (`S3DATA`)

| Backend | Port | Description |
|---------|------|-------------|
| `file` | 9991 | Local filesystem via `dataserver.js` |
| `multiple` | - | Multi-backend gateway (AWS S3, Azure, GCP, Sproxyd) |
| `mem` | - | In-memory (testing only) |

`scality` is an alias for `multiple`. With `multiple`, objects route to
backends defined in `locationConfig.json` based on location constraints.

### Metadata Backends (`S3METADATA`)

| Backend | Port | Description |
|---------|------|-------------|
| `file` (default) | 9990 | Local LevelDB via `mdserver.js` |
| `scality` | 9000 | External bucketd service (production Scality RING) |
| `mongodb` | 27017+ | MongoDB replica set |
| `mem` | - | In-memory (testing only) |

**file vs scality**: The `file` backend runs a self-contained metadata server
(`mdserver.js`) for development. The `scality` backend connects to external
**bucketd** which uses Raft-based distributed metadata (repd) for production
HA deployments.

### Auth Backends (`S3VAULT`)

| Backend | Port | Description |
|---------|------|-------------|
| `mem` (default) | - | In-memory accounts from `conf/authdata.json` |
| `vault` | 8500 | External Vault IAM service (vaultd) |

### KMS Backends (`S3KMS`)

| Backend | Description |
|---------|-------------|
| `file` (default) | Local file-based key storage |
| `mem` | In-memory (testing only) |
| `kmip` | External KMIP server |
| `aws` | AWS KMS |
| `scality` | Scality KMS |

### Path Configuration

- `S3DATAPATH`: Data storage directory (default: `./localData`)
- `S3METADATAPATH`: Metadata storage directory (default: `./localMetadata`)

### Key Files

- `config.json` - Default configuration
- `constants.js` - Global constants (splitters, limits, service accounts)
- `locationConfig.json` - Location constraints for multi-backend

## Testing Conventions

- **Framework**: Mocha with Sinon for mocking
- **Timeout**: 40000ms global
- **Coverage**: NYC (Istanbul), 80% patch target
- **Reporters**: mocha-multi-reporters (spec + junit XML)

Run a single test file:

```bash
S3BACKEND=mem yarn mocha tests/unit/path/to/test.js --exit
```

Run tests matching a pattern:

```bash
S3BACKEND=mem yarn mocha --grep "pattern" tests/unit --recursive --exit
```

## Code Style

- ESLint with `@scality/eslint-config-scality`
- `mocha/no-exclusive-tests: error` - Never commit `.only()` tests
- Many formatting rules disabled for legacy compatibility
- ECMAScript 2021, CommonJS modules (`sourceType: "script"`)
Loading