400 lines
11 KiB
Markdown
400 lines
11 KiB
Markdown
# @repo/storage
|
||
|
||
一个完全兼容 Hono 的存储解决方案，支持本地存储和 S3 兼容存储，提供 TUS 协议上传、文件管理和 REST API。
|
||
|
||
## 特性
|
||
|
||
- 🚀 **多存储支持**: 支持本地文件系统和 S3 兼容存储
|
||
- 📤 **TUS 协议**: 支持可恢复的文件上传
|
||
- 🔧 **Hono 集成**: 提供开箱即用的 Hono 中间件
|
||
- 📊 **文件管理**: 完整的文件生命周期管理
|
||
- 🗄️ **数据库集成**: 与 Prisma 深度集成
|
||
- ⏰ **自动清理**: 支持过期文件自动清理
|
||
- 🔄 **存储迁移**: 支持不同存储类型间的数据迁移
|
||
- 🔌 **适配器模式** - 通过适配器与任何数据库后端集成
|
||
- 📁 **多存储后端** - 支持本地存储和 S3 兼容存储
|
||
- 🚀 **TUS 协议** - 支持可恢复文件上传
|
||
- 🔄 **自动清理** - 自动清理失败的上传
|
||
- 🛡️ **类型安全** - 完整的 TypeScript 支持
|
||
|
||
## 安装
|
||
|
||
```bash
|
||
npm install @repo/storage
|
||
```
|
||
|
||
## 环境变量配置
|
||
|
||
### 基础配置
|
||
|
||
| 变量名 | 类型 | 默认值 | 描述 |
|
||
| ---------------------- | --------------- | ------- | ------------------------------------- |
|
||
| `STORAGE_TYPE` | `local` \| `s3` | `local` | 存储类型选择 |
|
||
| `UPLOAD_EXPIRATION_MS` | `number` | `0` | 上传文件过期时间（毫秒），0表示不过期 |
|
||
|
||
### 本地存储配置
|
||
|
||
当 `STORAGE_TYPE=local` 时需要配置：
|
||
|
||
| 变量名 | 类型 | 默认值 | 描述 |
|
||
| ------------ | -------- | ----------- | ---------------- |
|
||
| `UPLOAD_DIR` | `string` | `./uploads` | 本地存储目录路径 |
|
||
|
||
### S3 存储配置
|
||
|
||
当 `STORAGE_TYPE=s3` 时需要配置：
|
||
|
||
| 变量名 | 类型 | 默认值 | 描述 | 必需 |
|
||
| --------------------------- | --------- | ----------- | ---------------------------------- | ---- |
|
||
| `S3_BUCKET` | `string` | - | S3 存储桶名称 | ✅ |
|
||
| `S3_REGION` | `string` | `us-east-1` | S3 区域 | ✅ |
|
||
| `S3_ACCESS_KEY_ID` | `string` | - | S3 访问密钥 ID | ✅ |
|
||
| `S3_SECRET_ACCESS_KEY` | `string` | - | S3 访问密钥 | ✅ |
|
||
| `S3_ENDPOINT` | `string` | - | 自定义 S3 端点（用于兼容其他服务） | ❌ |
|
||
| `S3_FORCE_PATH_STYLE` | `boolean` | `false` | 是否强制使用路径样式 | ❌ |
|
||
| `S3_PART_SIZE` | `number` | `8388608` | 分片上传大小（8MB） | ❌ |
|
||
| `S3_MAX_CONCURRENT_UPLOADS` | `number` | `60` | 最大并发上传数 | ❌ |
|
||
|
||
## 配置示例
|
||
|
||
### 本地存储配置
|
||
|
||
```bash
|
||
# .env
|
||
STORAGE_TYPE=local
|
||
UPLOAD_DIR=./uploads
|
||
```
|
||
|
||
### AWS S3 配置
|
||
|
||
```bash
|
||
# .env
|
||
STORAGE_TYPE=s3
|
||
S3_BUCKET=my-app-uploads
|
||
S3_REGION=us-west-2
|
||
S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
|
||
S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
|
||
```
|
||
|
||
### MinIO 配置
|
||
|
||
```bash
|
||
# .env
|
||
STORAGE_TYPE=s3
|
||
S3_BUCKET=uploads
|
||
S3_REGION=us-east-1
|
||
S3_ACCESS_KEY_ID=minioadmin
|
||
S3_SECRET_ACCESS_KEY=minioadmin
|
||
S3_ENDPOINT=http://localhost:9000
|
||
S3_FORCE_PATH_STYLE=true
|
||
```
|
||
|
||
### 阿里云 OSS 配置
|
||
|
||
```bash
|
||
# .env
|
||
STORAGE_TYPE=s3
|
||
S3_BUCKET=my-oss-bucket
|
||
S3_REGION=oss-cn-hangzhou
|
||
S3_ACCESS_KEY_ID=your-access-key-id
|
||
S3_SECRET_ACCESS_KEY=your-access-key-secret
|
||
S3_ENDPOINT=https://oss-cn-hangzhou.aliyuncs.com
|
||
S3_FORCE_PATH_STYLE=false
|
||
```
|
||
|
||
### 腾讯云 COS 配置
|
||
|
||
```bash
|
||
# .env
|
||
STORAGE_TYPE=s3
|
||
S3_BUCKET=my-cos-bucket-1234567890
|
||
S3_REGION=ap-beijing
|
||
S3_ACCESS_KEY_ID=your-secret-id
|
||
S3_SECRET_ACCESS_KEY=your-secret-key
|
||
S3_ENDPOINT=https://cos.ap-beijing.myqcloud.com
|
||
S3_FORCE_PATH_STYLE=false
|
||
```
|
||
|
||
## 快速开始
|
||
|
||
### 1. 安装依赖
|
||
|
||
```bash
|
||
npm install @repo/storage
|
||
```
|
||
|
||
### 2. 实现数据库适配器
|
||
|
||
```typescript
|
||
import { DatabaseAdapter, ResourceData, CreateResourceData, StorageType } from '@repo/storage';
|
||
|
||
export class MyDatabaseAdapter implements DatabaseAdapter {
|
||
async getResourceByFileId(fileId: string): Promise<{ status: string; resource?: ResourceData }> {
|
||
// 实现从数据库获取资源的逻辑
|
||
}
|
||
|
||
async createResource(data: CreateResourceData): Promise<ResourceData> {
|
||
// 实现创建资源的逻辑
|
||
}
|
||
|
||
async updateResource(id: string, data: any): Promise<ResourceData> {
|
||
// 实现更新资源的逻辑
|
||
}
|
||
|
||
async deleteResource(id: string): Promise<ResourceData> {
|
||
// 实现删除资源的逻辑
|
||
}
|
||
|
||
async updateResourceStatus(fileId: string, status: string, additionalData?: any): Promise<ResourceData> {
|
||
// 实现更新资源状态的逻辑
|
||
}
|
||
|
||
async deleteFailedUploadingResource(expirationPeriod: number): Promise<{ count: number }> {
|
||
// 实现清理失败上传的逻辑
|
||
}
|
||
|
||
async migrateResourcesStorageType(
|
||
fromStorageType: StorageType,
|
||
toStorageType: StorageType,
|
||
): Promise<{ count: number }> {
|
||
// 实现存储类型迁移的逻辑
|
||
}
|
||
}
|
||
```
|
||
|
||
### 3. 注册适配器
|
||
|
||
```typescript
|
||
import { adapterRegistry } from '@repo/storage';
|
||
import { MyDatabaseAdapter } from './my-database-adapter';
|
||
|
||
// 在应用启动时注册适配器
|
||
adapterRegistry.setDatabaseAdapter(new MyDatabaseAdapter());
|
||
```
|
||
|
||
### 4. 使用存储功能
|
||
|
||
```typescript
|
||
import { createStorageApp, startCleanupScheduler } from '@repo/storage';
|
||
|
||
// 创建存储应用
|
||
const storageApp = createStorageApp({
|
||
apiBasePath: '/api/storage',
|
||
uploadPath: '/upload',
|
||
});
|
||
|
||
// 启动清理任务
|
||
startCleanupScheduler();
|
||
```
|
||
|
||
## Prisma 适配器示例
|
||
|
||
如果您使用 Prisma，可以参考以下实现：
|
||
|
||
```typescript
|
||
import { prisma } from '@your/db-package';
|
||
import { DatabaseAdapter, ResourceData, CreateResourceData, StorageType } from '@repo/storage';
|
||
|
||
export class PrismaDatabaseAdapter implements DatabaseAdapter {
|
||
// 将 Prisma Resource 转换为 ResourceData
|
||
private transformResource(resource: any): ResourceData {
|
||
return {
|
||
id: resource.id,
|
||
fileId: resource.fileId,
|
||
title: resource.title,
|
||
type: resource.type,
|
||
storageType: resource.storageType as StorageType,
|
||
status: resource.status || 'unknown',
|
||
meta: resource.meta,
|
||
createdAt: resource.createdAt,
|
||
updatedAt: resource.updatedAt,
|
||
};
|
||
}
|
||
|
||
async getResourceByFileId(fileId: string): Promise<{ status: string; resource?: ResourceData }> {
|
||
const resource = await prisma.resource.findFirst({
|
||
where: { fileId },
|
||
});
|
||
|
||
if (!resource) {
|
||
return { status: 'pending' };
|
||
}
|
||
|
||
return {
|
||
status: resource.status || 'unknown',
|
||
resource: this.transformResource(resource),
|
||
};
|
||
}
|
||
|
||
async createResource(data: CreateResourceData): Promise<ResourceData> {
|
||
const resource = await prisma.resource.create({
|
||
data: {
|
||
fileId: data.fileId,
|
||
title: data.filename,
|
||
type: data.mimeType,
|
||
storageType: data.storageType,
|
||
status: data.status || 'UPLOADING',
|
||
meta: {
|
||
size: data.size,
|
||
hash: data.hash,
|
||
},
|
||
},
|
||
});
|
||
return this.transformResource(resource);
|
||
}
|
||
|
||
// ... 实现其他方法
|
||
}
|
||
```
|
||
|
||
## API 参考
|
||
|
||
### DatabaseAdapter 接口
|
||
|
||
所有数据库适配器都必须实现 `DatabaseAdapter` 接口：
|
||
|
||
- `getResourceByFileId(fileId: string)` - 根据文件ID获取资源
|
||
- `createResource(data: CreateResourceData)` - 创建新资源
|
||
- `updateResource(id: string, data: any)` - 更新资源
|
||
- `deleteResource(id: string)` - 删除资源
|
||
- `updateResourceStatus(fileId: string, status: string, additionalData?: any)` - 更新资源状态
|
||
- `deleteFailedUploadingResource(expirationPeriod: number)` - 清理失败的上传
|
||
- `migrateResourcesStorageType(from: StorageType, to: StorageType)` - 迁移存储类型
|
||
|
||
### 适配器注册器
|
||
|
||
```typescript
|
||
import { adapterRegistry } from '@repo/storage';
|
||
|
||
// 注册适配器
|
||
adapterRegistry.setDatabaseAdapter(adapter);
|
||
|
||
// 获取当前适配器
|
||
const adapter = adapterRegistry.getDatabaseAdapter();
|
||
|
||
// 检查是否已注册适配器
|
||
const hasAdapter = adapterRegistry.hasAdapter();
|
||
```
|
||
|
||
## API 端点
|
||
|
||
### 文件资源管理
|
||
|
||
- `GET /api/storage/resource/:fileId` - 获取文件资源信息
|
||
- `GET /api/storage/resources` - 获取所有资源
|
||
- `GET /api/storage/resources/storage/:storageType` - 按存储类型获取资源
|
||
- `GET /api/storage/resources/status/:status` - 按状态获取资源
|
||
- `GET /api/storage/resources/uploading` - 获取正在上传的资源
|
||
- `DELETE /api/storage/resource/:id` - 删除资源
|
||
- `PATCH /api/storage/resource/:id` - 更新资源
|
||
|
||
### 文件访问和下载
|
||
|
||
- `GET /download/:fileId` - 文件下载和访问（支持所有存储类型）
|
||
|
||
### 统计和管理
|
||
|
||
- `GET /api/storage/stats` - 获取资源统计信息
|
||
- `POST /api/storage/cleanup` - 手动清理过期上传
|
||
- `POST /api/storage/cleanup/by-status` - 按状态清理资源
|
||
- `POST /api/storage/migrate-storage` - 迁移存储类型
|
||
|
||
### 存储配置
|
||
|
||
- `GET /api/storage/storage/info` - 获取存储信息
|
||
- `POST /api/storage/storage/switch` - 切换存储配置
|
||
- `POST /api/storage/storage/validate` - 验证存储配置
|
||
|
||
### 文件上传
|
||
|
||
- `POST /upload` - TUS 协议文件上传
|
||
- `PATCH /upload/:id` - 续传文件
|
||
- `HEAD /upload/:id` - 获取上传状态
|
||
|
||
## 数据库操作
|
||
|
||
```typescript
|
||
import {
|
||
getAllResources,
|
||
getResourceByFileId,
|
||
createResource,
|
||
updateResourceStatus,
|
||
deleteResource,
|
||
} from '@repo/storage';
|
||
|
||
// 获取所有资源
|
||
const resources = await getAllResources();
|
||
|
||
// 根据文件ID获取资源
|
||
const { status, resource } = await getResourceByFileId('file-id');
|
||
|
||
// 创建新资源
|
||
const newResource = await createResource({
|
||
fileId: 'unique-file-id',
|
||
filename: 'example.jpg',
|
||
size: 1024000,
|
||
mimeType: 'image/jpeg',
|
||
storageType: 'local',
|
||
});
|
||
```
|
||
|
||
## 文件生命周期
|
||
|
||
1. **上传开始**: 创建资源记录，状态为 `UPLOADING`
|
||
2. **上传完成**: 状态更新为 `UPLOADED`
|
||
3. **处理中**: 状态可更新为 `PROCESSING`
|
||
4. **处理完成**: 状态更新为 `PROCESSED`
|
||
5. **清理**: 过期文件自动清理
|
||
|
||
## 存储迁移
|
||
|
||
支持在不同存储类型之间迁移数据：
|
||
|
||
```bash
|
||
# API 调用示例
|
||
curl -X POST http://localhost:3000/api/storage/migrate-storage \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"from": "local", "to": "s3"}'
|
||
```
|
||
|
||
## 安全考虑
|
||
|
||
1. **环境变量**: 敏感信息（如 S3 密钥）应存储在环境变量中
|
||
2. **访问控制**: 建议在生产环境中添加适当的身份验证
|
||
3. **CORS 配置**: 根据需要配置跨域访问策略
|
||
4. **文件验证**: 建议添加文件类型和大小验证
|
||
|
||
## 故障排除
|
||
|
||
### 常见问题
|
||
|
||
1. **找不到模块错误**: 确保已正确安装依赖包
|
||
2. **S3 连接失败**: 检查网络连接和凭据配置
|
||
3. **本地存储权限**: 确保应用有写入本地目录的权限
|
||
4. **上传失败**: 检查文件大小限制和存储空间
|
||
|
||
### 调试模式
|
||
|
||
启用详细日志：
|
||
|
||
```bash
|
||
DEBUG=storage:* npm start
|
||
```
|
||
|
||
## 许可证
|
||
|
||
MIT License
|
||
|
||
## 贡献
|
||
|
||
欢迎提交 Issue 和 Pull Request！
|
||
|
||
## 更新日志
|
||
|
||
### v2.0.0
|
||
|
||
- 重构为模块化架构
|
||
- 添加完整的 TypeScript 支持
|
||
- 支持多种 S3 兼容服务
|
||
- 改进的错误处理和日志记录
|