diff --git a/.cursorignore b/.cursorignore index 6f9f00f..56662bb 100644 --- a/.cursorignore +++ b/.cursorignore @@ -1 +1,2 @@ # Add directories or file patterns to ignore during indexing (e.g. foo/ or *.csv) +# *.env \ No newline at end of file diff --git a/README.md b/README.md index 7fd254f..047a08a 100644 --- a/README.md +++ b/README.md @@ -4,22 +4,164 @@ This template is for creating a monorepo with Turborepo, shadcn/ui, tailwindcss v4, and react v19. +## 项目结构 + +``` +├── apps/ +│ ├── backend/ # Hono 后端应用 +│ └── web/ # Next.js 前端应用 +├── packages/ +│ ├── db/ # Prisma 数据库包 +│ ├── storage/ # 存储解决方案包 +│ ├── tus/ # TUS 上传协议包 +│ └── ui/ # UI 组件包 +└── docs/ # 文档 +``` + +## 特性 + +- 🚀 **现代技术栈**: Next.js 15, React 19, Hono, Prisma +- 📦 **Monorepo**: 使用 Turborepo 管理多包项目 +- 🎨 **UI 组件**: shadcn/ui + TailwindCSS v4 +- 📤 **文件上传**: 支持 TUS 协议的可恢复上传 +- 💾 **多存储支持**: 本地存储 + S3 兼容存储 +- 🗄️ **数据库**: PostgreSQL + Prisma ORM +- 🔄 **实时通信**: WebSocket 支持 + +## 快速开始 + +### 1. 安装依赖 + +```bash +pnpm install +``` + +### 2. 环境变量配置 + +复制环境变量模板并配置: + +```bash +cp .env.example .env +``` + +#### 存储配置 + +**本地存储(开发环境推荐):** + +```bash +STORAGE_TYPE=local +UPLOAD_DIR=./uploads +``` + +**S3 存储(生产环境推荐):** + +```bash +STORAGE_TYPE=s3 +S3_BUCKET=your-bucket-name +S3_REGION=us-east-1 +S3_ACCESS_KEY_ID=your-access-key +S3_SECRET_ACCESS_KEY=your-secret-key +``` + +**MinIO 本地开发:** + +```bash +STORAGE_TYPE=s3 +S3_BUCKET=uploads +S3_ENDPOINT=http://localhost:9000 +S3_ACCESS_KEY_ID=minioadmin +S3_SECRET_ACCESS_KEY=minioadmin +S3_FORCE_PATH_STYLE=true +``` + +详细的环境变量配置请参考:[环境变量配置指南](./docs/ENVIRONMENT.md) + +### 3. 数据库设置 + +```bash +# 生成 Prisma 客户端 +pnpm db:generate + +# 运行数据库迁移 +pnpm db:migrate + +# 填充种子数据(可选) +pnpm db:seed +``` + +### 4. 启动开发服务器 + +```bash +pnpm dev +``` + +这将启动: + +- 前端应用: http://localhost:3001 +- 后端 API: http://localhost:3000 +- 文件上传: http://localhost:3000/upload +- 存储管理 API: http://localhost:3000/api/storage + +## 存储包 (@repo/storage) + +项目包含一个功能完整的存储解决方案包,支持: + +### 核心功能 + +- 🗂️ **多存储后端**: 本地文件系统、AWS S3、MinIO、阿里云 OSS、腾讯云 COS +- 📤 **TUS 上传**: 支持可恢复的大文件上传 +- 🔧 **Hono 集成**: 提供即插即用的中间件 +- 📊 **文件管理**: 完整的文件生命周期管理 +- ⏰ **自动清理**: 过期文件自动清理机制 +- 🔄 **存储迁移**: 支持不同存储类型间的数据迁移 + +### API 端点 + +```bash +# 文件资源管理 +GET /api/storage/resources # 获取所有资源 +GET /api/storage/resource/:fileId # 获取文件信息 +DELETE /api/storage/resource/:id # 删除资源 + +# 文件访问和下载 +GET /download/:fileId # 文件下载和访问(支持所有存储类型) + +# 统计和管理 +GET /api/storage/stats # 获取统计信息 +POST /api/storage/cleanup # 清理过期文件 +POST /api/storage/migrate-storage # 迁移存储类型 + +# 文件上传 (TUS 协议) +POST /upload # 开始上传 +PATCH /upload/:id # 续传文件 +HEAD /upload/:id # 获取上传状态 +``` + +### 使用示例 + +```typescript +import { createStorageApp, startCleanupScheduler } from '@repo/storage'; + +// 创建存储应用 +const storageApp = createStorageApp({ + apiBasePath: '/api/storage', + uploadPath: '/upload', +}); + +// 挂载到主应用 +app.route('/', storageApp); + +// 启动清理调度器 +startCleanupScheduler(); +``` + ## One-click Deploy You can deploy this template to Vercel with the button below: -[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?build-command=cd+..%2F..%2F+%26%26+pnpm+turbo+build+--filter%3Dweb...&demo-description=This+is+a+template+Turborepo+with+ShadcnUI+tailwindv4&demo-image=%2F%2Fimages.ctfassets.net%2Fe5382hct74si%2F2JxNyYATuuV7WPuJ31kF9Q%2F433990aa4c8e7524a9095682fb08f0b1%2FBasic.png&demo-title=Turborepo+%26+Next.js+Starter&demo-url=https%3A%2F%2Fexamples-basic-web.vercel.sh%2F&from=templates&project-name=Turborepo+%26+Next.js+Starter&repository-name=turborepo-shadcn-tailwind&repository-url=https%3A%2F%2Fgithub.com%2Flinkb15%2Fturborepo-shadcn-ui-tailwind-4&root-directory=apps%2Fweb&skippable-integrations=1) +[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?build-command=cd+..%2F..%2F+%26%26+pnpm+turbo+build+--filter%3Dweb...&demo-description=This+is+a+template+Turborepo+with+ShadcnUI+tailwindv4&demo-image=%2F%2Fimages.ctfassets.net%2Fe5382hct74si%2F2JxNyYATuuV7WPuJ31kF9Q%2F433990aa4c8e7524a9095682fb08f0b1%2FBasic.png&demo-title=Turborepo+%26+Next.js+Starter&demo-url=https%3A%2F%2Fexamples-basic-web.vercel.sh%2F&from=templates&project-name=Turborepo+%26+Next.js+Starter&repository-name=turborepo-shadcn-tailwind&repository-url=https%3A%2F%2Flinkb15%2Fturborepo-shadcn-ui-tailwind-4&root-directory=apps%2Fweb&skippable-integrations=1) -## Usage - -in the root directory run: - -```bash -pnpm install -pnpm dev -``` - -## Adding components +## 添加 UI 组件 To add components to your app, run the following command at the root of your `web` app: @@ -33,7 +175,7 @@ This will place the ui components in the `packages/ui/src/components` directory. Your `globals.css` are already set up to use the components from the `ui` package which is imported in the `web` app. -## Using components +## 使用组件 To use the components in your app, import them from the `ui` package. @@ -41,11 +183,44 @@ To use the components in your app, import them from the `ui` package. import { Button } from '@repo/ui/components/ui/button'; ``` +## 脚本命令 + +```bash +# 开发 +pnpm dev # 启动所有应用 +pnpm dev:web # 只启动前端 +pnpm dev:backend # 只启动后端 + +# 构建 +pnpm build # 构建所有包 +pnpm build:web # 构建前端 +pnpm build:backend # 构建后端 + +# 数据库 +pnpm db:generate # 生成 Prisma 客户端 +pnpm db:migrate # 运行数据库迁移 +pnpm db:seed # 填充种子数据 +pnpm db:studio # 打开 Prisma Studio + +# 代码质量 +pnpm lint # 代码检查 +pnpm type-check # 类型检查 +pnpm format # 代码格式化 +``` + +## 文档 + +- [环境变量配置指南](./docs/ENVIRONMENT.md) +- [存储包文档](./packages/storage/README.md) +- [文件访问使用指南](./docs/STATIC_FILES.md) + ## More Resources - [shadcn/ui - Monorepo](https://ui.shadcn.com/docs/monorepo) - [Turborepo - shadcn/ui](https://turbo.build/repo/docs/guides/tools/shadcn-ui) - [TailwindCSS v4 - Explicitly Registering Sources](https://tailwindcss.com/docs/detecting-classes-in-source-files#explicitly-registering-sources) +- [Hono Documentation](https://hono.dev/) +- [TUS Protocol](https://tus.io/) [opengraph-image]: https://turborepo-shadcn-tailwind.vercel.app/opengraph-image.png [opengraph-image-url]: https://turborepo-shadcn-tailwind.vercel.app/ diff --git a/apps/backend/package.json b/apps/backend/package.json index 4b7406c..1e898bf 100644 --- a/apps/backend/package.json +++ b/apps/backend/package.json @@ -1,32 +1,33 @@ { - "name": "backend", - "scripts": { - "dev": "bun run --hot src/index.ts" - }, - "dependencies": { - "@elastic/elasticsearch": "^9.0.2", - "@hono/node-server": "^1.14.3", - "@hono/trpc-server": "^0.3.4", - "@hono/zod-validator": "^0.5.0", - "@repo/db": "workspace:*", - "@repo/oidc-provider": "workspace:*", - "@trpc/server": "11.1.2", - "hono": "^4.7.10", - "ioredis": "5.4.1", - "jose": "^6.0.11", - "minio": "7.1.3", - "nanoid": "^5.1.5", - "node-cron": "^4.0.7", - "oidc-provider": "^9.1.1", - "superjson": "^2.2.2", - "valibot": "^1.1.0", - "zod": "^3.25.23" - }, - "devDependencies": { - "@types/bun": "latest", - "@types/node": "^22.15.21", - "@types/oidc-provider": "^9.1.0", - "supertest": "^7.1.1", - "vitest": "^3.1.4" - } + "name": "backend", + "scripts": { + "dev": "bun run --hot src/index.ts" + }, + "dependencies": { + "@elastic/elasticsearch": "^9.0.2", + "@hono/node-server": "^1.14.3", + "@hono/trpc-server": "^0.3.4", + "@hono/zod-validator": "^0.5.0", + "@repo/db": "workspace:*", + "@repo/oidc-provider": "workspace:*", + "@trpc/server": "11.1.2", + "hono": "^4.7.10", + "ioredis": "5.4.1", + "jose": "^6.0.11", + "minio": "7.1.3", + "nanoid": "^5.1.5", + "node-cron": "^4.0.7", + "oidc-provider": "^9.1.1", + "superjson": "^2.2.2", + "valibot": "^1.1.0", + "zod": "^3.25.23", + "@repo/storage": "workspace:*" + }, + "devDependencies": { + "@types/bun": "latest", + "@types/node": "^22.15.21", + "@types/oidc-provider": "^9.1.0", + "supertest": "^7.1.1", + "vitest": "^3.1.4" + } } diff --git a/apps/backend/src/index.ts b/apps/backend/src/index.ts index 4c70029..fe1390c 100644 --- a/apps/backend/src/index.ts +++ b/apps/backend/src/index.ts @@ -17,8 +17,8 @@ import { wsHandler, wsConfig } from './socket'; // 导入新的路由 import userRest from './user/user.rest'; -import uploadRest from './upload/upload.rest'; -import { startCleanupScheduler } from './upload/scheduler'; +// 使用新的 @repo/storage 包 +import { createStorageApp, startCleanupScheduler } from '@repo/storage'; type Env = { Variables: { @@ -58,9 +58,13 @@ app.use( // 添加 REST API 路由 app.route('/api/users', userRest); -app.route('/api/upload', uploadRest); - +// 使用新的存储应用,包含API和上传功能 +const storageApp = createStorageApp({ + apiBasePath: '/api/storage', + uploadPath: '/upload', +}); +app.route('/', storageApp); // 添加 WebSocket 路由 app.get('/ws', wsHandler); diff --git a/apps/backend/src/upload/README.md b/apps/backend/src/upload/README.md deleted file mode 100644 index 38aac07..0000000 --- a/apps/backend/src/upload/README.md +++ /dev/null @@ -1,232 +0,0 @@ -# 上传模块架构改造 - -本模块已从 NestJS 架构成功改造为 Hono + Bun 架构,并支持多种存储后端的无感切换。 - -## 文件结构 - -``` -src/upload/ -├── tus.ts # TUS 协议服务核心实现 -├── upload.index.ts # 资源管理相关函数 -├── upload.rest.ts # Hono REST API 路由 -├── storage.adapter.ts # 存储适配器系统 🆕 -├── storage.utils.ts # 存储工具类 🆕 -├── scheduler.ts # 定时清理任务 -├── utils.ts # 工具函数 -├── types.ts # 类型定义 -└── README.md # 本文档 -``` - -## 存储适配器系统 - -### 支持的存储类型 - -1. **本地存储 (Local)** - 文件存储在本地文件系统 -2. **S3 存储 (S3)** - 文件存储在 AWS S3 或兼容的对象存储服务 - -### 环境变量配置 - -#### 本地存储配置 - -```bash -STORAGE_TYPE=local -UPLOAD_DIR=./uploads -UPLOAD_EXPIRATION_MS=0 # 0 表示不自动过期(推荐设置) -``` - -#### S3 存储配置 - -```bash -STORAGE_TYPE=s3 -S3_BUCKET=your-bucket-name -S3_REGION=us-east-1 -S3_ACCESS_KEY_ID=your-access-key -S3_SECRET_ACCESS_KEY=your-secret-key -S3_ENDPOINT=https://s3.amazonaws.com # 可选,支持其他 S3 兼容服务 -S3_FORCE_PATH_STYLE=false # 可选,路径风格 -S3_PART_SIZE=8388608 # 可选,分片大小 (8MB) -S3_MAX_CONCURRENT_UPLOADS=60 # 可选,最大并发上传数 -UPLOAD_EXPIRATION_MS=0 # 0 表示不自动过期(推荐设置) -``` - -### 存储类型记录 - -- **数据库支持**: 每个资源记录都包含 `storageType` 字段,标识文件使用的存储后端 -- **自动记录**: 上传时自动记录当前的存储类型 -- **迁移支持**: 支持批量更新现有资源的存储类型标记 - -### 不过期设置 - -- **默认行为**: 过期时间默认设为 0,表示文件不会自动过期 -- **手动清理**: 提供多种手动清理选项 -- **灵活控制**: 可根据需要设置过期时间,或完全禁用自动清理 - -### 无感切换机制 - -1. **单例模式管理**: `StorageManager` 使用单例模式确保全局一致性 -2. **自动配置检测**: 启动时根据环境变量自动选择存储类型 -3. **统一接口**: 所有存储类型都实现相同的 TUS `DataStore` 接口 -4. **运行时切换**: 支持运行时切换存储配置(需要重启生效) - -## API 端点 - -### 资源管理 - -- `GET /api/upload/resource/:fileId` - 获取文件资源信息 -- `GET /api/upload/resources` - 获取所有资源 -- `GET /api/upload/resources/storage/:storageType` - 🆕 根据存储类型获取资源 -- `GET /api/upload/resources/status/:status` - 🆕 根据状态获取资源 -- `GET /api/upload/resources/uploading` - 🆕 获取正在上传的资源 -- `GET /api/upload/stats` - 🆕 获取资源统计信息 -- `DELETE /api/upload/resource/:id` - 删除资源 -- `PATCH /api/upload/resource/:id` - 更新资源 -- `POST /api/upload/cleanup` - 手动触发清理 -- `POST /api/upload/cleanup/by-status` - 🆕 根据状态清理资源 -- `POST /api/upload/migrate-storage` - 🆕 迁移资源存储类型标记 - -### 存储管理 - -- `GET /api/upload/storage/info` - 获取当前存储配置信息 -- `POST /api/upload/storage/switch` - 切换存储类型 -- `POST /api/upload/storage/validate` - 验证存储配置 - -### TUS 协议 - -- `OPTIONS /api/upload/*` - TUS 协议选项请求 -- `HEAD /api/upload/*` - TUS 协议头部请求 -- `POST /api/upload/*` - TUS 协议创建上传 -- `PATCH /api/upload/*` - TUS 协议上传数据 -- `GET /api/upload/*` - TUS 协议获取状态 - -## 新增 API 使用示例 - -### 获取存储类型统计 - -```javascript -const response = await fetch('/api/upload/stats'); -const stats = await response.json(); -// { -// total: 150, -// byStatus: { "UPLOADED": 120, "UPLOADING": 5, "PROCESSED": 25 }, -// byStorageType: { "local": 80, "s3": 70 } -// } -``` - -### 查询特定存储类型的资源 - -```javascript -const response = await fetch('/api/upload/resources/storage/s3'); -const s3Resources = await response.json(); -``` - -### 迁移存储类型标记 - -```javascript -const response = await fetch('/api/upload/migrate-storage', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ from: 'local', to: 's3' }), -}); -// { success: true, message: "Migrated 50 resources from local to s3", count: 50 } -``` - -### 手动清理特定状态的资源 - -```javascript -const response = await fetch('/api/upload/cleanup/by-status', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ - status: 'UPLOADING', - olderThanDays: 7, - }), -}); -``` - -## 🆕 存储管理示例 - -### 获取存储信息 - -```javascript -const response = await fetch('/api/upload/storage/info'); -const storageInfo = await response.json(); -// { type: 'local', config: { directory: './uploads' } } -``` - -### 切换到 S3 存储 - -```javascript -const newConfig = { - type: 's3', - s3: { - bucket: 'my-bucket', - region: 'us-west-2', - accessKeyId: 'YOUR_ACCESS_KEY', - secretAccessKey: 'YOUR_SECRET_KEY', - }, -}; - -const response = await fetch('/api/upload/storage/switch', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(newConfig), -}); -``` - -### 验证存储配置 - -```javascript -const response = await fetch('/api/upload/storage/validate', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(newConfig), -}); -const validation = await response.json(); -// { valid: true, message: 'Storage configuration is valid' } -``` - -## 特性保留 - -1. **TUS 协议支持** - 完全保留原有的断点续传功能 -2. **文件命名** - 保留安全的文件命名策略 -3. **资源状态管理** - 保留完整的上传状态跟踪 -4. **自动清理** - 保留过期文件清理功能(默认禁用) -5. **数据库集成** - 保留 Prisma ORM 数据库操作 - -## 🆕 新增特性 - -1. **多存储后端支持** - 支持本地存储和 S3 存储 -2. **无感切换** - 运行时可切换存储类型 -3. **配置验证** - 提供存储配置验证功能 -4. **存储信息查询** - 可查询当前存储配置 -5. **统一日志** - 存储操作统一日志记录 -6. **🆕 存储类型记录** - 数据库记录每个资源的存储类型 -7. **🆕 灵活清理** - 支持按状态、时间等条件清理 -8. **🆕 统计分析** - 提供详细的资源统计信息 -9. **🆕 不过期设置** - 默认不自动过期,避免意外删除 - -## 运行 - -服务启动时会自动: - -1. 根据环境变量初始化存储适配器 -2. 初始化 TUS 服务器 -3. 注册 REST API 路由 -4. 启动定时清理任务(如果启用) - -支持的存储切换场景: - -- 开发环境使用本地存储 -- 生产环境使用 S3 存储 -- 混合云部署灵活切换 -- 存储迁移时批量更新资源标记 - -## 💡 最佳实践 - -1. **过期设置**: 推荐设置 `UPLOAD_EXPIRATION_MS=0` 避免文件意外过期 -2. **存储记录**: 利用数据库中的 `storageType` 字段追踪文件位置 -3. **定期清理**: 使用手动清理 API 定期清理不需要的资源 -4. **监控统计**: 使用统计 API 监控存储使用情况 -5. **迁移策略**: 在存储迁移时先更新环境变量,再使用迁移 API 更新数据库标记 - -无需代码修改,仅通过环境变量即可实现存储后端的无感切换。 diff --git a/apps/backend/src/upload/types.ts b/apps/backend/src/upload/types.ts deleted file mode 100644 index 2140ebc..0000000 --- a/apps/backend/src/upload/types.ts +++ /dev/null @@ -1,29 +0,0 @@ -export interface UploadCompleteEvent { - identifier: string; - filename: string; - size: number; - hash: string; - integrityVerified: boolean; -} - -export type UploadEvent = { - uploadStart: { - identifier: string; - filename: string; - totalSize: number; - resuming?: boolean; - }; - uploadComplete: UploadCompleteEvent; - uploadError: { identifier: string; error: string; filename: string }; -}; -export interface UploadLock { - clientId: string; - timestamp: number; -} -// 添加重试机制,处理临时网络问题 -// 实现定期清理过期的临时文件 -// 添加文件完整性校验 -// 实现上传进度持久化,支持服务重启后恢复 -// 添加并发限制,防止系统资源耗尽 -// 实现文件去重功能,避免重复上传 -// 添加日志记录和监控机制 diff --git a/apps/backend/src/upload/upload.rest.ts b/apps/backend/src/upload/upload.rest.ts deleted file mode 100644 index 440e457..0000000 --- a/apps/backend/src/upload/upload.rest.ts +++ /dev/null @@ -1,198 +0,0 @@ -import { Hono } from 'hono'; -import { handleTusRequest, cleanupExpiredUploads, getStorageInfo } from './tus'; -import { - getResourceByFileId, - getAllResources, - deleteResource, - updateResource, - getResourcesByStorageType, - getResourcesByStatus, - getUploadingResources, - getResourceStats, - migrateResourcesStorageType, -} from './upload.index'; -import { StorageManager, StorageType, type StorageConfig } from './storage.adapter'; -import { prisma } from '@repo/db'; - -const uploadRest = new Hono(); - -// 获取文件资源信息 -uploadRest.get('/resource/:fileId', async (c) => { - const fileId = c.req.param('fileId'); - const result = await getResourceByFileId(fileId); - return c.json(result); -}); - -// 获取所有资源 -uploadRest.get('/resources', async (c) => { - const resources = await getAllResources(); - return c.json(resources); -}); - -// 根据存储类型获取资源 -uploadRest.get('/resources/storage/:storageType', async (c) => { - const storageType = c.req.param('storageType') as StorageType; - const resources = await getResourcesByStorageType(storageType); - return c.json(resources); -}); - -// 根据状态获取资源 -uploadRest.get('/resources/status/:status', async (c) => { - const status = c.req.param('status'); - const resources = await getResourcesByStatus(status); - return c.json(resources); -}); - -// 获取正在上传的资源 -uploadRest.get('/resources/uploading', async (c) => { - const resources = await getUploadingResources(); - return c.json(resources); -}); - -// 获取资源统计信息 -uploadRest.get('/stats', async (c) => { - const stats = await getResourceStats(); - return c.json(stats); -}); - -// 删除资源 -uploadRest.delete('/resource/:id', async (c) => { - const id = c.req.param('id'); - const result = await deleteResource(id); - return c.json(result); -}); - -// 更新资源 -uploadRest.patch('/resource/:id', async (c) => { - const id = c.req.param('id'); - const data = await c.req.json(); - const result = await updateResource(id, data); - return c.json(result); -}); - -// 迁移资源存储类型(批量更新数据库中的存储类型标记) -uploadRest.post('/migrate-storage', async (c) => { - try { - const { from, to } = await c.req.json(); - const result = await migrateResourcesStorageType(from as StorageType, to as StorageType); - return c.json({ - success: true, - message: `Migrated ${result.count} resources from ${from} to ${to}`, - count: result.count, - }); - } catch (error) { - console.error('Failed to migrate storage type:', error); - return c.json( - { - success: false, - error: error instanceof Error ? error.message : 'Unknown error', - }, - 400, - ); - } -}); - -// 清理过期上传 -uploadRest.post('/cleanup', async (c) => { - const result = await cleanupExpiredUploads(); - return c.json(result); -}); - -// 手动清理指定状态的资源 -uploadRest.post('/cleanup/by-status', async (c) => { - try { - const { status, olderThanDays } = await c.req.json(); - const cutoffDate = new Date(); - cutoffDate.setDate(cutoffDate.getDate() - (olderThanDays || 30)); - - const deletedResources = await prisma.resource.deleteMany({ - where: { - status, - createdAt: { - lt: cutoffDate, - }, - }, - }); - - return c.json({ - success: true, - message: `Deleted ${deletedResources.count} resources with status ${status}`, - count: deletedResources.count, - }); - } catch (error) { - console.error('Failed to cleanup by status:', error); - return c.json( - { - success: false, - error: error instanceof Error ? error.message : 'Unknown error', - }, - 400, - ); - } -}); - -// 获取存储信息 -uploadRest.get('/storage/info', async (c) => { - const storageInfo = getStorageInfo(); - return c.json(storageInfo); -}); - -// 切换存储类型(需要重启应用) -uploadRest.post('/storage/switch', async (c) => { - try { - const newConfig = (await c.req.json()) as StorageConfig; - const storageManager = StorageManager.getInstance(); - await storageManager.switchStorage(newConfig); - - return c.json({ - success: true, - message: 'Storage configuration updated. Please restart the application for changes to take effect.', - newType: newConfig.type, - }); - } catch (error) { - console.error('Failed to switch storage:', error); - return c.json( - { - success: false, - error: error instanceof Error ? error.message : 'Unknown error', - }, - 400, - ); - } -}); - -// 验证存储配置 -uploadRest.post('/storage/validate', async (c) => { - try { - const config = (await c.req.json()) as StorageConfig; - const { validateStorageConfig } = await import('./storage.adapter'); - const errors = validateStorageConfig(config); - - if (errors.length > 0) { - return c.json({ valid: false, errors }, 400); - } - - return c.json({ valid: true, message: 'Storage configuration is valid' }); - } catch (error) { - return c.json( - { - valid: false, - errors: [error instanceof Error ? error.message : 'Invalid JSON'], - }, - 400, - ); - } -}); - -// TUS 协议处理 - 使用通用处理器 -uploadRest.all('/*', async (c) => { - try { - await handleTusRequest(c.req.raw, c.res); - return new Response(null); - } catch (error) { - console.error('TUS request error:', error); - return c.json({ error: 'Upload request failed' }, 500); - } -}); - -export default uploadRest; diff --git a/apps/backend/src/upload/utils.ts b/apps/backend/src/upload/utils.ts deleted file mode 100644 index a7c189f..0000000 --- a/apps/backend/src/upload/utils.ts +++ /dev/null @@ -1,4 +0,0 @@ -export function extractFileIdFromNginxUrl(url: string) { - const match = url.match(/uploads\/(\d{4}\/\d{2}\/\d{2}\/[^/]+)/); - return match ? match[1] : ''; -} diff --git a/apps/backend/tsconfig.json b/apps/backend/tsconfig.json index 53412b3..ed8e081 100644 --- a/apps/backend/tsconfig.json +++ b/apps/backend/tsconfig.json @@ -4,7 +4,8 @@ "moduleResolution": "bundler", "paths": { "@/*": ["./*"], - "@repo/db/*": ["../../packages/db/src/*"] + "@repo/db/*": ["../../packages/db/src/*"], + "@repo/storage/*": ["../../packages/storage/src/*"] } } } diff --git a/apps/web/app/upload/page.tsx b/apps/web/app/upload/page.tsx new file mode 100644 index 0000000..cef182f --- /dev/null +++ b/apps/web/app/upload/page.tsx @@ -0,0 +1,44 @@ +'use client'; +import { FileUpload } from '../../components/FileUpload'; +import { FileDownload } from '../../components/FileDownload'; +import { AdvancedFileDownload } from '../../components/AdvancedFileDownload'; +import { DownloadTester } from '../../components/DownloadTester'; + +export default function UploadPage() { + return ( +
+
+
+

文件上传和下载中心

+

完整的文件管理解决方案:上传、下载、预览

+
+ + {/* 上传组件 */} +
+

📤 文件上传

+ +
+ + {/* 下载测试组件 */} +
+

🔧 下载测试

+ +
+ +
+ {/* 基础下载组件 */} +
+

📥 基础下载

+ +
+ + {/* 高级下载组件 */} +
+

🚀 高级下载

+ +
+
+
+
+ ); +} diff --git a/apps/web/components/AdvancedFileDownload.tsx b/apps/web/components/AdvancedFileDownload.tsx new file mode 100644 index 0000000..34ceeac --- /dev/null +++ b/apps/web/components/AdvancedFileDownload.tsx @@ -0,0 +1,234 @@ +import React, { useState } from 'react'; +import { useFileDownload } from '../hooks/useFileDownload'; +import { useTusUpload } from '../hooks/useTusUpload'; + +export function AdvancedFileDownload() { + const { getFileInfo } = useTusUpload(); + const { + downloadProgress, + isDownloading, + downloadError, + downloadFile, + downloadFileWithProgress, + previewFile, + copyFileLink, + canPreview, + getFileIcon, + } = useFileDownload(); + + const [fileId, setFileId] = useState(''); + const [fileInfo, setFileInfo] = useState(null); + const [loading, setLoading] = useState(false); + const [error, setError] = useState(null); + + // 获取文件信息 + const handleGetFileInfo = async () => { + if (!fileId.trim()) { + setError('请输入文件ID'); + return; + } + + setLoading(true); + setError(null); + + try { + const info = await getFileInfo(fileId); + if (info) { + setFileInfo(info); + } else { + setError('文件不存在或未准备好'); + } + } catch (err) { + setError('获取文件信息失败'); + } finally { + setLoading(false); + } + }; + + // 简单下载 + const handleSimpleDownload = () => { + downloadFile(fileId, fileInfo?.title); + }; + + // 带进度的下载 + const handleProgressDownload = async () => { + try { + await downloadFileWithProgress(fileId, fileInfo?.title); + } catch (error) { + console.error('Download with progress failed:', error); + } + }; + + // 预览文件 + const handlePreview = () => { + previewFile(fileId); + }; + + // 复制链接 + const handleCopyLink = async () => { + try { + await copyFileLink(fileId); + alert('链接已复制到剪贴板!'); + } catch (error) { + alert('复制失败'); + } + }; + + return ( +
+

高级文件下载

+ + {/* 文件ID输入 */} +
+ +
+ setFileId(e.target.value)} + placeholder="输入文件ID" + className="flex-1 px-3 py-2 border border-gray-300 rounded-md focus:outline-none focus:ring-2 focus:ring-blue-500" + /> + +
+
+ + {/* 错误信息 */} + {(error || downloadError) && ( +
+ {error || downloadError} +
+ )} + + {/* 下载进度 */} + {isDownloading && downloadProgress && ( +
+
+ 下载进度 + {downloadProgress.percentage}% +
+
+
+
+
+ {formatFileSize(downloadProgress.loaded)} / {formatFileSize(downloadProgress.total)} +
+
+ )} + + {/* 文件信息 */} + {fileInfo && ( +
+
+ {getFileIcon(fileInfo.type || '')} +
+

{fileInfo.title || '未知文件'}

+

{fileInfo.type || '未知类型'}

+
+
+ +
+
+ 状态: + + {fileInfo.status || '未知'} + +
+ {fileInfo.meta?.size && ( +
+ 大小: {formatFileSize(fileInfo.meta.size)} +
+ )} +
+ 创建时间: {new Date(fileInfo.createdAt).toLocaleString()} +
+
+ 存储类型: {fileInfo.storageType || '未知'} +
+
+
+ )} + + {/* 操作按钮 */} + {fileInfo && ( +
+
+ + + {canPreview(fileInfo.type || '') && ( + + )} + +
+ + {/* 文件预览提示 */} + {canPreview(fileInfo.type || '') && ( +
+

💡 此文件支持在线预览,点击"预览文件"可以在浏览器中直接查看

+
+ )} +
+ )} + + {/* 使用说明 */} +
+

功能说明:

+
    +
  • + • 快速下载:直接通过浏览器下载文件 +
  • +
  • + • 进度下载:显示下载进度,适合大文件 +
  • +
  • + • 预览文件:图片、PDF、视频等可在线预览 +
  • +
  • + • 复制链接:复制文件访问链接到剪贴板 +
  • +
+
+
+ ); +} + +// 格式化文件大小 +function formatFileSize(bytes: number): string { + if (bytes === 0) return '0 Bytes'; + const k = 1024; + const sizes = ['Bytes', 'KB', 'MB', 'GB', 'TB']; + const i = Math.floor(Math.log(bytes) / Math.log(k)); + return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i]; +} diff --git a/apps/web/components/DownloadTester.tsx b/apps/web/components/DownloadTester.tsx new file mode 100644 index 0000000..aa472b2 --- /dev/null +++ b/apps/web/components/DownloadTester.tsx @@ -0,0 +1,90 @@ +import React, { useState } from 'react'; +import { useTusUpload } from '../hooks/useTusUpload'; + +export function DownloadTester() { + const { serverUrl, getFileInfo } = useTusUpload(); + const [fileId, setFileId] = useState('2025/05/28/1mVGC8r6jy'); + const [testResults, setTestResults] = useState(null); + const [loading, setLoading] = useState(false); + + const runTests = async () => { + setLoading(true); + const results: any = { + fileId, + serverUrl, + timestamp: new Date().toISOString(), + }; + + try { + // 测试1: 检查资源信息 + console.log('Testing resource info...'); + const resourceInfo = await getFileInfo(fileId); + results.resourceInfo = resourceInfo; + + // 测试2: 测试下载端点 + console.log('Testing download endpoint...'); + const downloadUrl = `${serverUrl}/download/${fileId}`; + results.downloadUrl = downloadUrl; + + const response = await fetch(downloadUrl, { method: 'HEAD' }); + results.downloadResponse = { + status: response.status, + statusText: response.statusText, + headers: Object.fromEntries(response.headers.entries()), + }; + + // 测试3: 测试API端点 + console.log('Testing API endpoint...'); + const apiUrl = `${serverUrl}/api/storage/resource/${fileId}`; + results.apiUrl = apiUrl; + + const apiResponse = await fetch(apiUrl); + const apiData = await apiResponse.json(); + results.apiResponse = { + status: apiResponse.status, + data: apiData, + }; + } catch (error) { + results.error = error instanceof Error ? error.message : String(error); + } + + setTestResults(results); + setLoading(false); + }; + + return ( +
+

🔧 下载功能测试

+ +
+ +
+ setFileId(e.target.value)} + className="flex-1 px-3 py-2 border border-gray-300 rounded-md focus:outline-none focus:ring-2 focus:ring-blue-500" + /> + +
+
+ + {testResults && ( +
+
+

测试结果

+
+							{JSON.stringify(testResults, null, 2)}
+						
+
+
+ )} +
+ ); +} diff --git a/apps/web/components/FileDownload.tsx b/apps/web/components/FileDownload.tsx new file mode 100644 index 0000000..3e9b544 --- /dev/null +++ b/apps/web/components/FileDownload.tsx @@ -0,0 +1,157 @@ +import React, { useState } from 'react'; +import { useTusUpload } from '../hooks/useTusUpload'; + +interface FileDownloadProps { + fileId?: string; + fileName?: string; + className?: string; +} + +export function FileDownload({ fileId, fileName, className }: FileDownloadProps) { + const { getFileUrlByFileId, getFileInfo } = useTusUpload(); + const [inputFileId, setInputFileId] = useState(fileId || ''); + const [fileInfo, setFileInfo] = useState(null); + const [loading, setLoading] = useState(false); + const [error, setError] = useState(null); + + // 获取文件信息 + const handleGetFileInfo = async () => { + if (!inputFileId.trim()) { + setError('请输入文件ID'); + return; + } + + setLoading(true); + setError(null); + + try { + const info = await getFileInfo(inputFileId); + if (info) { + setFileInfo(info); + } else { + setError('文件不存在或未准备好'); + } + } catch (err) { + setError('获取文件信息失败'); + } finally { + setLoading(false); + } + }; + + // 直接下载文件 + const handleDirectDownload = () => { + const downloadUrl = getFileUrlByFileId(inputFileId); + window.open(downloadUrl, '_blank'); + }; + + // 复制下载链接 + const handleCopyLink = async () => { + const downloadUrl = getFileUrlByFileId(inputFileId); + try { + await navigator.clipboard.writeText(downloadUrl); + alert('下载链接已复制到剪贴板!'); + } catch (error) { + console.error('复制失败:', error); + } + }; + + // 在新窗口预览文件 + const handlePreview = () => { + const downloadUrl = getFileUrlByFileId(inputFileId); + window.open(downloadUrl, '_blank'); + }; + + return ( +
+

文件下载

+ + {/* 文件ID输入 */} +
+ +
+ setInputFileId(e.target.value)} + placeholder="输入文件ID" + className="flex-1 px-3 py-2 border border-gray-300 rounded-md focus:outline-none focus:ring-2 focus:ring-blue-500" + /> + +
+
+ + {/* 错误信息 */} + {error &&
{error}
} + + {/* 文件信息 */} + {fileInfo && ( +
+

文件信息

+
+

+ 文件名: {fileInfo.title || '未知'} +

+

+ 类型: {fileInfo.type || '未知'} +

+

+ 状态: {fileInfo.status || '未知'} +

+ {fileInfo.meta?.size && ( +

+ 大小: {formatFileSize(fileInfo.meta.size)} +

+ )} +

+ 创建时间: {new Date(fileInfo.createdAt).toLocaleString()} +

+
+
+ )} + + {/* 操作按钮 */} + {inputFileId && ( +
+ + + +
+ )} + + {/* 使用说明 */} +
+

使用说明:

+
    +
  • • 输入文件ID后点击"查询"获取文件信息
  • +
  • • "直接下载"会打开新窗口下载文件
  • +
  • • "预览/查看"适用于图片、PDF等可预览的文件
  • +
  • • "复制链接"将下载地址复制到剪贴板
  • +
+
+
+ ); +} + +// 格式化文件大小 +function formatFileSize(bytes: number): string { + if (bytes === 0) return '0 Bytes'; + const k = 1024; + const sizes = ['Bytes', 'KB', 'MB', 'GB', 'TB']; + const i = Math.floor(Math.log(bytes) / Math.log(k)); + return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i]; +} diff --git a/apps/web/components/FileUpload.tsx b/apps/web/components/FileUpload.tsx new file mode 100644 index 0000000..ce8670d --- /dev/null +++ b/apps/web/components/FileUpload.tsx @@ -0,0 +1,218 @@ +'use client'; +import React, { useCallback, useState } from 'react'; +import { useTusUpload } from '../hooks/useTusUpload'; + +interface UploadedFile { + fileId: string; + fileName: string; + url: string; +} + +export function FileUpload() { + const { uploadProgress, isUploading, uploadError, handleFileUpload, getFileUrlByFileId, serverUrl } = useTusUpload(); + + const [uploadedFiles, setUploadedFiles] = useState([]); + const [dragOver, setDragOver] = useState(false); + + // 处理文件选择 + const handleFileSelect = useCallback( + async (files: FileList | null) => { + if (!files || files.length === 0) return; + + for (let i = 0; i < files.length; i++) { + const file = files[i]; + + try { + const result = await handleFileUpload( + file, + (result) => { + console.log('Upload success:', result); + setUploadedFiles((prev) => [ + ...prev, + { + fileId: result.fileId, + fileName: result.fileName, + url: result.url, + }, + ]); + }, + (error) => { + console.error('Upload error:', error); + }, + ); + } catch (error) { + console.error('Upload failed:', error); + } + } + }, + [handleFileUpload], + ); + + // 处理拖拽上传 + const handleDrop = useCallback( + (e: React.DragEvent) => { + e.preventDefault(); + setDragOver(false); + handleFileSelect(e.dataTransfer.files); + }, + [handleFileSelect], + ); + + const handleDragOver = useCallback((e: React.DragEvent) => { + e.preventDefault(); + setDragOver(true); + }, []); + + const handleDragLeave = useCallback((e: React.DragEvent) => { + e.preventDefault(); + setDragOver(false); + }, []); + + // 处理文件输入 + const handleInputChange = useCallback( + (e: React.ChangeEvent) => { + handleFileSelect(e.target.files); + }, + [handleFileSelect], + ); + + // 复制链接到剪贴板 + const copyToClipboard = useCallback(async (url: string) => { + try { + await navigator.clipboard.writeText(url); + alert('链接已复制到剪贴板!'); + } catch (error) { + console.error('Failed to copy:', error); + } + }, []); + + return ( +
+

文件上传

+ + {/* 服务器信息 */} +
+

+ 服务器地址: {serverUrl} +

+
+ + {/* 拖拽上传区域 */} +
+
+
+ + + +
+
+

拖拽文件到这里,或者

+ +
+

支持多文件上传,TUS 协议支持断点续传

+
+
+ + {/* 上传进度 */} + {isUploading && ( +
+
+ 上传中... + {uploadProgress}% +
+
+
+
+
+ )} + + {/* 错误信息 */} + {uploadError && ( +
+

+ 上传失败: + {uploadError} +

+
+ )} + + {/* 已上传文件列表 */} + {uploadedFiles.length > 0 && ( +
+

已上传文件

+
+ {uploadedFiles.map((file, index) => ( +
+
+
+ + + +
+
+

{file.fileName}

+

文件ID: {file.fileId}

+
+
+
+ + 查看 + + +
+
+ ))} +
+
+ )} + + {/* 使用说明 */} +
+

使用说明:

+
    +
  • • 支持拖拽和点击上传
  • +
  • • 使用 TUS 协议,支持大文件和断点续传
  • +
  • • 上传完成后可以通过链接直接访问文件
  • +
  • • 图片和 PDF 会在浏览器中直接显示
  • +
  • • 其他文件类型会触发下载
  • +
+
+
+ ); +} diff --git a/apps/web/components/SimpleUploadExample.tsx b/apps/web/components/SimpleUploadExample.tsx new file mode 100644 index 0000000..f5eb1d9 --- /dev/null +++ b/apps/web/components/SimpleUploadExample.tsx @@ -0,0 +1,75 @@ +import React, { useState } from 'react'; +import { useTusUpload } from '../hooks/useTusUpload'; + +export function SimpleUploadExample() { + const { uploadProgress, isUploading, uploadError, handleFileUpload, getFileUrlByFileId } = useTusUpload(); + const [uploadedFileUrl, setUploadedFileUrl] = useState(''); + + const handleFileChange = async (e: React.ChangeEvent) => { + const file = e.target.files?.[0]; + if (!file) return; + + try { + const result = await handleFileUpload( + file, + (result) => { + console.log('上传成功!', result); + setUploadedFileUrl(result.url); + }, + (error) => { + console.error('上传失败:', error); + }, + ); + } catch (error) { + console.error('上传出错:', error); + } + }; + + return ( +
+

简单上传示例

+ +
+ +
+ + {isUploading && ( +
+
+ 上传进度 + {uploadProgress}% +
+
+
+
+
+ )} + + {uploadError && ( +
{uploadError}
+ )} + + {uploadedFileUrl && ( +
+

上传成功!

+ + 查看文件 + +
+ )} +
+ ); +} diff --git a/apps/web/docs/UPLOAD_HOOK_USAGE.md b/apps/web/docs/UPLOAD_HOOK_USAGE.md new file mode 100644 index 0000000..c3f27c0 --- /dev/null +++ b/apps/web/docs/UPLOAD_HOOK_USAGE.md @@ -0,0 +1,287 @@ +# TUS 上传 Hook 使用指南 + +## 概述 + +`useTusUpload` 是一个自定义 React Hook,提供了基于 TUS 协议的文件上传功能,支持大文件上传、断点续传、进度跟踪等特性。 + +## 环境变量配置 + +确保在 `.env` 文件中配置了以下环境变量: + +```env +NEXT_PUBLIC_SERVER_PORT=3000 +NEXT_PUBLIC_SERVER_IP=http://localhost +``` + +**注意**:在 Next.js 中,客户端组件只能访问以 `NEXT_PUBLIC_` 开头的环境变量。 + +## Hook API + +### 返回值 + +```typescript +const { + uploadProgress, // 上传进度 (0-100) + isUploading, // 是否正在上传 + uploadError, // 上传错误信息 + handleFileUpload, // 文件上传函数 + getFileUrlByFileId, // 根据文件ID获取访问链接 + getFileInfo, // 获取文件详细信息 + getUploadStatus, // 获取上传状态 + serverUrl, // 服务器地址 +} = useTusUpload(); +``` + +### 主要方法 + +#### `handleFileUpload(file, onSuccess?, onError?)` + +上传文件的主要方法。 + +**参数:** + +- `file: File` - 要上传的文件对象 +- `onSuccess?: (result: UploadResult) => void` - 成功回调 +- `onError?: (error: string) => void` - 失败回调 + +**返回:** `Promise` + +**UploadResult 接口:** + +```typescript +interface UploadResult { + compressedUrl: string; // 压缩版本URL(当前与原始URL相同) + url: string; // 文件访问URL + fileId: string; // 文件唯一标识 + fileName: string; // 文件名 +} +``` + +#### `getFileUrlByFileId(fileId: string)` + +根据文件ID生成访问链接。 + +**参数:** + +- `fileId: string` - 文件唯一标识 + +**返回:** `string` - 文件访问URL + +## 使用示例 + +### 基础使用 + +```tsx +import React, { useState } from 'react'; +import { useTusUpload } from '../hooks/useTusUpload'; + +function UploadComponent() { + const { uploadProgress, isUploading, uploadError, handleFileUpload } = useTusUpload(); + const [uploadedUrl, setUploadedUrl] = useState(''); + + const handleFileChange = async (e: React.ChangeEvent) => { + const file = e.target.files?.[0]; + if (!file) return; + + try { + const result = await handleFileUpload( + file, + (result) => { + console.log('上传成功!', result); + setUploadedUrl(result.url); + }, + (error) => { + console.error('上传失败:', error); + }, + ); + } catch (error) { + console.error('上传出错:', error); + } + }; + + return ( +
+ + + {isUploading && ( +
+

上传进度: {uploadProgress}%

+ +
+ )} + + {uploadError &&

{uploadError}

} + + {uploadedUrl && ( + + 查看上传的文件 + + )} +
+ ); +} +``` + +### 拖拽上传 + +```tsx +import React, { useCallback, useState } from 'react'; +import { useTusUpload } from '../hooks/useTusUpload'; + +function DragDropUpload() { + const { handleFileUpload, isUploading, uploadProgress } = useTusUpload(); + const [dragOver, setDragOver] = useState(false); + + const handleDrop = useCallback( + async (e: React.DragEvent) => { + e.preventDefault(); + setDragOver(false); + + const files = e.dataTransfer.files; + if (files.length > 0) { + await handleFileUpload(files[0]); + } + }, + [handleFileUpload], + ); + + const handleDragOver = useCallback((e: React.DragEvent) => { + e.preventDefault(); + setDragOver(true); + }, []); + + return ( +
setDragOver(false)} + style={{ + border: dragOver ? '2px dashed #0070f3' : '2px dashed #ccc', + padding: '20px', + textAlign: 'center', + backgroundColor: dragOver ? '#f0f8ff' : '#fafafa', + }} + > + {isUploading ?

上传中... {uploadProgress}%

:

拖拽文件到这里上传

} +
+ ); +} +``` + +### 多文件上传 + +```tsx +function MultiFileUpload() { + const { handleFileUpload } = useTusUpload(); + const [uploadingFiles, setUploadingFiles] = useState>(new Map()); + + const handleFilesChange = async (e: React.ChangeEvent) => { + const files = e.target.files; + if (!files) return; + + for (let i = 0; i < files.length; i++) { + const file = files[i]; + const fileId = `${file.name}-${Date.now()}-${i}`; + + setUploadingFiles((prev) => new Map(prev).set(fileId, 0)); + + try { + await handleFileUpload( + file, + (result) => { + console.log(`文件 ${file.name} 上传成功:`, result); + setUploadingFiles((prev) => { + const newMap = new Map(prev); + newMap.delete(fileId); + return newMap; + }); + }, + (error) => { + console.error(`文件 ${file.name} 上传失败:`, error); + setUploadingFiles((prev) => { + const newMap = new Map(prev); + newMap.delete(fileId); + return newMap; + }); + }, + ); + } catch (error) { + console.error(`文件 ${file.name} 上传出错:`, error); + } + } + }; + + return ( +
+ + + {uploadingFiles.size > 0 && ( +
+

正在上传的文件:

+ {Array.from(uploadingFiles.entries()).map(([fileId, progress]) => ( +
+ {fileId}: {progress}% +
+ ))} +
+ )} +
+ ); +} +``` + +## 特性 + +### 1. 断点续传 + +TUS 协议支持断点续传,如果上传过程中断,可以从中断的地方继续上传。 + +### 2. 大文件支持 + +适合上传大文件,没有文件大小限制(取决于服务器配置)。 + +### 3. 进度跟踪 + +实时显示上传进度,提供良好的用户体验。 + +### 4. 错误处理 + +提供详细的错误信息和重试机制。 + +### 5. 自动重试 + +内置重试机制,网络异常时自动重试。 + +## 故障排除 + +### 1. 环境变量获取不到 + +确保环境变量以 `NEXT_PUBLIC_` 开头,并且 Next.js 应用已重启。 + +### 2. 上传失败 + +检查服务器是否正在运行,端口是否正确。 + +### 3. CORS 错误 + +确保后端服务器配置了正确的 CORS 设置。 + +### 4. 文件无法访问 + +确认文件上传成功后,检查返回的 URL 是否正确。 + +## 注意事项 + +1. **Next.js 环境变量**:客户端组件只能访问 `NEXT_PUBLIC_` 前缀的环境变量 +2. **服务器配置**:确保后端服务器支持 TUS 协议 +3. **文件大小**:虽然支持大文件,但要注意服务器和客户端的内存限制 +4. **网络环境**:在网络不稳定的环境下,断点续传功能特别有用 + +## API 路由 + +Hook 会访问以下 API 路由: + +- `POST /upload` - TUS 上传端点 +- `GET /download/:fileId` - 文件下载/访问 +- `GET /api/storage/resource/:fileId` - 获取文件信息 +- `HEAD /upload/:fileId` - 获取上传状态 diff --git a/apps/web/hooks/useFileDownload.ts b/apps/web/hooks/useFileDownload.ts new file mode 100644 index 0000000..4379a4b --- /dev/null +++ b/apps/web/hooks/useFileDownload.ts @@ -0,0 +1,180 @@ +import { useState } from 'react'; +import { useTusUpload } from './useTusUpload'; + +interface DownloadProgress { + loaded: number; + total: number; + percentage: number; +} + +export function useFileDownload() { + const { getFileUrlByFileId, serverUrl } = useTusUpload(); + const [downloadProgress, setDownloadProgress] = useState(null); + const [isDownloading, setIsDownloading] = useState(false); + const [downloadError, setDownloadError] = useState(null); + + // 直接下载文件(浏览器处理) + const downloadFile = (fileId: string, filename?: string) => { + const url = getFileUrlByFileId(fileId); + const link = document.createElement('a'); + link.href = url; + if (filename) { + link.download = filename; + } + link.target = '_blank'; + document.body.appendChild(link); + link.click(); + document.body.removeChild(link); + }; + + // 带进度的文件下载 + const downloadFileWithProgress = async ( + fileId: string, + filename?: string, + onProgress?: (progress: DownloadProgress) => void, + ): Promise => { + return new Promise(async (resolve, reject) => { + setIsDownloading(true); + setDownloadError(null); + setDownloadProgress(null); + + try { + const url = getFileUrlByFileId(fileId); + const response = await fetch(url); + + if (!response.ok) { + throw new Error(`HTTP error! status: ${response.status}`); + } + + const contentLength = response.headers.get('Content-Length'); + const total = contentLength ? parseInt(contentLength, 10) : 0; + let loaded = 0; + + const reader = response.body?.getReader(); + if (!reader) { + throw new Error('Failed to get response reader'); + } + + const chunks: Uint8Array[] = []; + + while (true) { + const { done, value } = await reader.read(); + + if (done) break; + + if (value) { + chunks.push(value); + loaded += value.length; + + const progress = { + loaded, + total, + percentage: total > 0 ? Math.round((loaded / total) * 100) : 0, + }; + + setDownloadProgress(progress); + onProgress?.(progress); + } + } + + // 创建 Blob + const blob = new Blob(chunks); + + // 如果提供了文件名,自动下载 + if (filename) { + const downloadUrl = URL.createObjectURL(blob); + const link = document.createElement('a'); + link.href = downloadUrl; + link.download = filename; + document.body.appendChild(link); + link.click(); + document.body.removeChild(link); + URL.revokeObjectURL(downloadUrl); + } + + setIsDownloading(false); + setDownloadProgress(null); + resolve(blob); + } catch (error) { + const errorMessage = error instanceof Error ? error.message : 'Download failed'; + setDownloadError(errorMessage); + setIsDownloading(false); + setDownloadProgress(null); + reject(new Error(errorMessage)); + } + }); + }; + + // 预览文件(在新窗口打开) + const previewFile = (fileId: string) => { + const url = getFileUrlByFileId(fileId); + window.open(url, '_blank', 'noopener,noreferrer'); + }; + + // 获取文件的 Blob URL(用于预览) + const getFileBlobUrl = async (fileId: string): Promise => { + try { + const blob = await downloadFileWithProgress(fileId); + return URL.createObjectURL(blob); + } catch (error) { + throw new Error('Failed to create blob URL'); + } + }; + + // 复制文件链接到剪贴板 + const copyFileLink = async (fileId: string): Promise => { + try { + const url = getFileUrlByFileId(fileId); + await navigator.clipboard.writeText(url); + } catch (error) { + throw new Error('Failed to copy link'); + } + }; + + // 检查文件是否可以预览(基于 MIME 类型) + const canPreview = (mimeType: string): boolean => { + const previewableTypes = [ + 'image/', // 所有图片 + 'application/pdf', + 'text/', + 'video/', + 'audio/', + ]; + + return previewableTypes.some((type) => mimeType.startsWith(type)); + }; + + // 获取文件类型图标 + const getFileIcon = (mimeType: string): string => { + if (mimeType.startsWith('image/')) return '🖼️'; + if (mimeType.startsWith('video/')) return '🎥'; + if (mimeType.startsWith('audio/')) return '🎵'; + if (mimeType === 'application/pdf') return '📄'; + if (mimeType.startsWith('text/')) return '📝'; + if (mimeType.includes('word')) return '📝'; + if (mimeType.includes('excel') || mimeType.includes('spreadsheet')) return '📊'; + if (mimeType.includes('powerpoint') || mimeType.includes('presentation')) return '📊'; + if (mimeType.includes('zip') || mimeType.includes('rar') || mimeType.includes('archive')) return '📦'; + return '📁'; + }; + + return { + // 状态 + downloadProgress, + isDownloading, + downloadError, + + // 方法 + downloadFile, + downloadFileWithProgress, + previewFile, + getFileBlobUrl, + copyFileLink, + + // 工具函数 + canPreview, + getFileIcon, + getFileUrlByFileId, + serverUrl, + }; +} diff --git a/apps/web/hooks/useTusUpload.ts b/apps/web/hooks/useTusUpload.ts index 85c3d31..11d8c01 100644 --- a/apps/web/hooks/useTusUpload.ts +++ b/apps/web/hooks/useTusUpload.ts @@ -1,7 +1,5 @@ -import { useState } from "react"; -import * as tus from "tus-js-client"; -import { env } from "../env"; -import { getCompressedImageUrl } from "@nice/utils"; +import { useState } from 'react'; +import * as tus from 'tus-js-client'; interface UploadResult { compressedUrl: string; @@ -11,105 +9,146 @@ interface UploadResult { } export function useTusUpload() { - const [uploadProgress, setUploadProgress] = useState< - Record - >({}); - const [isUploading, setIsUploading] = useState(false); + const [uploadProgress, setUploadProgress] = useState(0); + const [isUploading, setIsUploading] = useState(false); const [uploadError, setUploadError] = useState(null); - const getFileId = (url: string) => { - const parts = url.split("/"); - const uploadIndex = parts.findIndex((part) => part === "upload"); - if (uploadIndex === -1 || uploadIndex + 4 >= parts.length) { - throw new Error("Invalid upload URL format"); - } - return parts.slice(uploadIndex + 1, uploadIndex + 5).join("/"); + // 获取服务器配置 + const getServerUrl = () => { + const ip = process.env.NEXT_PUBLIC_SERVER_IP || 'http://localhost'; + const port = process.env.NEXT_PUBLIC_SERVER_PORT || '3000'; + return `${ip}:${port}`; }; - const getResourceUrl = (url: string) => { - const parts = url.split("/"); - const uploadIndex = parts.findIndex((part) => part === "upload"); - if (uploadIndex === -1 || uploadIndex + 4 >= parts.length) { - throw new Error("Invalid upload URL format"); - } - const resUrl = `http://${env.SERVER_IP}:${env.FILE_PORT}/uploads/${parts.slice(uploadIndex + 1, uploadIndex + 6).join("/")}`; - return resUrl; - }; - const handleFileUpload = async ( - file: File | Blob, - onSuccess: (result: UploadResult) => void, - onError: (error: Error) => void, - fileKey: string // 添加文件唯一标识 - ) => { - // console.log() - setIsUploading(true); - setUploadProgress((prev) => ({ ...prev, [fileKey]: 0 })); - setUploadError(null); - try { - // 如果是Blob,需要转换为File - let fileName = "uploaded-file"; - if (file instanceof Blob && !(file instanceof File)) { - // 根据MIME类型设置文件扩展名 - const extension = file.type.split('/')[1]; - fileName = `uploaded-file.${extension}`; - } - const uploadFile = file instanceof Blob && !(file instanceof File) - ? new File([file], fileName, { type: file.type }) - : file as File; - console.log(`http://${env.SERVER_IP}:${env.SERVER_PORT}/upload`); - const upload = new tus.Upload(uploadFile, { - endpoint: `http://${env.SERVER_IP}:${env.SERVER_PORT}/upload`, - retryDelays: [0, 1000, 3000, 5000], + // 文件上传函数 + const handleFileUpload = ( + file: File, + onSuccess?: (result: UploadResult) => void, + onError?: (error: string) => void, + ): Promise => { + return new Promise((resolve, reject) => { + setIsUploading(true); + setUploadProgress(0); + setUploadError(null); + + const serverUrl = getServerUrl(); + const uploadUrl = `${serverUrl}/upload`; + + const upload = new tus.Upload(file, { + endpoint: uploadUrl, + retryDelays: [0, 3000, 5000, 10000, 20000], metadata: { - filename: uploadFile.name, - filetype: uploadFile.type, - size: uploadFile.size as any, - }, - onProgress: (bytesUploaded, bytesTotal) => { - const progress = Number( - ((bytesUploaded / bytesTotal) * 100).toFixed(2) - ); - setUploadProgress((prev) => ({ - ...prev, - [fileKey]: progress, - })); - }, - onSuccess: async (payload) => { - if (upload.url) { - const fileId = getFileId(upload.url); - //console.log(fileId) - const url = getResourceUrl(upload.url); - setIsUploading(false); - setUploadProgress((prev) => ({ - ...prev, - [fileKey]: 100, - })); - onSuccess({ - compressedUrl: getCompressedImageUrl(url), - url, - fileId, - fileName: uploadFile.name, - }); - } + filename: file.name, + filetype: file.type, }, onError: (error) => { - const err = - error instanceof Error - ? error - : new Error("Unknown error"); + console.error('Upload failed:', error); + const errorMessage = error.message || 'Upload failed'; + setUploadError(errorMessage); setIsUploading(false); - setUploadError(error.message); - console.log(error); - onError(err); + onError?.(errorMessage); + reject(new Error(errorMessage)); + }, + onProgress: (bytesUploaded, bytesTotal) => { + const percentage = Math.round((bytesUploaded / bytesTotal) * 100); + setUploadProgress(percentage); + }, + onSuccess: () => { + console.log('Upload completed successfully'); + setIsUploading(false); + setUploadProgress(100); + + // 从上传 URL 中提取目录格式的 fileId + const uploadUrl = upload.url; + if (!uploadUrl) { + const error = 'Failed to get upload URL'; + setUploadError(error); + onError?.(error); + reject(new Error(error)); + return; + } + + // 提取完整的上传ID,然后移除文件名部分得到目录路径 + const fullUploadId = uploadUrl.replace(/^.*\/upload\//, ''); + const fileId = fullUploadId.replace(/\/[^/]+$/, ''); + + console.log('Full upload ID:', fullUploadId); + console.log('Extracted fileId (directory):', fileId); + + const result: UploadResult = { + fileId, + fileName: file.name, + url: getFileUrlByFileId(fileId), + compressedUrl: getFileUrlByFileId(fileId), // 对于简单实现,压缩版本和原版本相同 + }; + + onSuccess?.(result); + resolve(result); }, }); + + // 开始上传 upload.start(); + }); + }; + + // 根据 fileId 获取文件访问链接 + const getFileUrlByFileId = (fileId: string): string => { + const serverUrl = getServerUrl(); + // 对fileId进行URL编码以处理其中的斜杠 + const encodedFileId = encodeURIComponent(fileId); + return `${serverUrl}/download/${encodedFileId}`; + }; + + // 检查文件是否存在并获取详细信息 + const getFileInfo = async (fileId: string) => { + try { + const serverUrl = getServerUrl(); + // 对fileId进行URL编码以处理其中的斜杠 + const encodedFileId = encodeURIComponent(fileId); + const response = await fetch(`${serverUrl}/api/storage/resource/${encodedFileId}`); + const data = await response.json(); + + if (data.status === 'UPLOADED' && data.resource) { + return { + ...data.resource, + url: getFileUrlByFileId(fileId), + }; + } + + console.log('File info response:', data); + return null; } catch (error) { - const err = - error instanceof Error ? error : new Error("Upload failed"); - setIsUploading(false); - setUploadError(err.message); - onError(err); + console.error('Failed to get file info:', error); + return null; + } + }; + + // 获取上传状态 + const getUploadStatus = async (fileId: string) => { + try { + const serverUrl = getServerUrl(); + const response = await fetch(`${serverUrl}/upload/${fileId}`, { + method: 'HEAD', + }); + + if (response.status === 200) { + const uploadLength = response.headers.get('Upload-Length'); + const uploadOffset = response.headers.get('Upload-Offset'); + + return { + isComplete: uploadLength === uploadOffset, + progress: + uploadLength && uploadOffset ? Math.round((parseInt(uploadOffset) / parseInt(uploadLength)) * 100) : 0, + uploadLength: uploadLength ? parseInt(uploadLength) : 0, + uploadOffset: uploadOffset ? parseInt(uploadOffset) : 0, + }; + } + + return null; + } catch (error) { + console.error('Failed to get upload status:', error); + return null; } }; @@ -118,5 +157,9 @@ export function useTusUpload() { isUploading, uploadError, handleFileUpload, + getFileUrlByFileId, + getFileInfo, + getUploadStatus, + serverUrl: getServerUrl(), }; } diff --git a/apps/web/lib/oidc-config.ts b/apps/web/lib/oidc-config.ts index 09ba761..ce35fc3 100644 --- a/apps/web/lib/oidc-config.ts +++ b/apps/web/lib/oidc-config.ts @@ -1,18 +1,26 @@ import { UserManager, WebStorageStateStore } from 'oidc-client-ts'; +// 创建存储配置的函数,避免 SSR 问题 +const createUserStore = () => { + if (typeof window !== 'undefined' && window.localStorage) { + return new WebStorageStateStore({ store: window.localStorage }); + } + return undefined; +}; + // OIDC 客户端配置 export const oidcConfig = { - authority: 'http://localhost:3000/oidc', // 后端OIDC provider地址 - client_id: 'demo-client', - client_secret: 'demo-client-secret', - redirect_uri: 'http://localhost:3001/auth/callback', - post_logout_redirect_uri: 'http://localhost:3001', - response_type: 'code', - scope: 'openid profile email', - automaticSilentRenew: true, - includeIdTokenInSilentRenew: true, - revokeTokensOnSignout: true, - userStore: new WebStorageStateStore({ store: window?.localStorage }), + authority: 'http://localhost:3000/oidc', // 后端OIDC provider地址 + client_id: 'demo-client', + client_secret: 'demo-client-secret', + redirect_uri: 'http://localhost:3001/auth/callback', + post_logout_redirect_uri: 'http://localhost:3001', + response_type: 'code', + scope: 'openid profile email', + automaticSilentRenew: true, + includeIdTokenInSilentRenew: true, + revokeTokensOnSignout: true, + ...(typeof window !== 'undefined' && { userStore: createUserStore() }), }; // 创建用户管理器实例 @@ -20,8 +28,8 @@ export const userManager = typeof window !== 'undefined' ? new UserManager(oidcC // OIDC 相关的URL export const oidcUrls = { - login: `${oidcConfig.authority}/auth`, - logout: `${oidcConfig.authority}/logout`, - token: `${oidcConfig.authority}/token`, - userinfo: `${oidcConfig.authority}/userinfo`, -}; \ No newline at end of file + login: `${oidcConfig.authority}/auth`, + logout: `${oidcConfig.authority}/logout`, + token: `${oidcConfig.authority}/token`, + userinfo: `${oidcConfig.authority}/userinfo`, +}; diff --git a/apps/web/package.json b/apps/web/package.json index 78e928e..cde8104 100644 --- a/apps/web/package.json +++ b/apps/web/package.json @@ -31,6 +31,7 @@ "react": "^19.1.0", "react-dom": "^19.1.0", "superjson": "^2.2.2", + "tus-js-client": "^4.3.1", "valibot": "^1.1.0" }, "devDependencies": { diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md new file mode 100644 index 0000000..d9c25ec --- /dev/null +++ b/docs/ENVIRONMENT.md @@ -0,0 +1,237 @@ +# 环境变量配置指南 + +本文档详细说明了项目中所有环境变量的配置方法和用途。 + +## 存储配置 (@repo/storage) + +### 基础配置 + +```bash +# 存储类型选择 +STORAGE_TYPE=local # 可选值: local | s3 + +# 上传文件过期时间(毫秒),0表示不过期 +UPLOAD_EXPIRATION_MS=0 +``` + +### 本地存储配置 + +当 `STORAGE_TYPE=local` 时需要配置: + +```bash +# 本地存储目录路径 +UPLOAD_DIR=./uploads +``` + +### S3 存储配置 + +当 `STORAGE_TYPE=s3` 时需要配置: + +```bash +# S3 存储桶名称 (必需) +S3_BUCKET=my-app-uploads + +# S3 区域 (必需) +S3_REGION=us-east-1 + +# S3 访问密钥 ID (必需) +S3_ACCESS_KEY_ID=your-access-key-id + +# S3 访问密钥 (必需) +S3_SECRET_ACCESS_KEY=your-secret-access-key + +# 自定义 S3 端点 (可选,用于 MinIO、阿里云 OSS 等) +S3_ENDPOINT= + +# 是否强制使用路径样式 (可选) +S3_FORCE_PATH_STYLE=false + +# 分片上传大小,单位字节 (可选,默认 8MB) +S3_PART_SIZE=8388608 + +# 最大并发上传数 (可选) +S3_MAX_CONCURRENT_UPLOADS=60 +``` + +## 配置示例 + +### 开发环境 - 本地存储 + +```bash +# .env.development +STORAGE_TYPE=local +UPLOAD_DIR=./uploads +UPLOAD_EXPIRATION_MS=86400000 # 24小时过期 +``` + +### 生产环境 - AWS S3 + +```bash +# .env.production +STORAGE_TYPE=s3 +S3_BUCKET=prod-app-uploads +S3_REGION=us-west-2 +S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE +S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +UPLOAD_EXPIRATION_MS=604800000 # 7天过期 +``` + +### MinIO 本地开发 + +```bash +# .env.local +STORAGE_TYPE=s3 +S3_BUCKET=uploads +S3_REGION=us-east-1 +S3_ACCESS_KEY_ID=minioadmin +S3_SECRET_ACCESS_KEY=minioadmin +S3_ENDPOINT=http://localhost:9000 +S3_FORCE_PATH_STYLE=true +``` + +### 阿里云 OSS + +```bash +# .env.aliyun +STORAGE_TYPE=s3 +S3_BUCKET=my-oss-bucket +S3_REGION=oss-cn-hangzhou +S3_ACCESS_KEY_ID=your-access-key-id +S3_SECRET_ACCESS_KEY=your-access-key-secret +S3_ENDPOINT=https://oss-cn-hangzhou.aliyuncs.com +S3_FORCE_PATH_STYLE=false +``` + +### 腾讯云 COS + +```bash +# .env.tencent +STORAGE_TYPE=s3 +S3_BUCKET=my-cos-bucket-1234567890 +S3_REGION=ap-beijing +S3_ACCESS_KEY_ID=your-secret-id +S3_SECRET_ACCESS_KEY=your-secret-key +S3_ENDPOINT=https://cos.ap-beijing.myqcloud.com +S3_FORCE_PATH_STYLE=false +``` + +## 其他配置 + +### 数据库配置 + +```bash +# PostgreSQL 数据库连接字符串 +DATABASE_URL="postgresql://username:password@localhost:5432/database" +``` + +### Redis 配置 + +```bash +# Redis 连接字符串 +REDIS_URL="redis://localhost:6379" +``` + +### 应用配置 + +```bash +# 应用端口 +PORT=3000 + +# 应用环境 +NODE_ENV=development + +# CORS 允许的源 +CORS_ORIGIN=http://localhost:3001 +``` + +## 安全注意事项 + +1. **敏感信息保护**: + + - 永远不要将包含敏感信息的 `.env` 文件提交到版本控制系统 + - 使用 `.env.example` 文件作为模板 + +2. **生产环境**: + + - 使用环境变量管理服务(如 AWS Secrets Manager、Azure Key Vault) + - 定期轮换访问密钥 + +3. **权限控制**: + - S3 存储桶应配置适当的访问策略 + - 使用最小权限原则 + +## 验证配置 + +可以使用以下 API 端点验证存储配置: + +```bash +# 验证存储配置 +curl -X POST http://localhost:3000/api/storage/storage/validate \ + -H "Content-Type: application/json" \ + -d '{ + "type": "s3", + "s3": { + "bucket": "my-bucket", + "region": "us-east-1", + "accessKeyId": "your-key", + "secretAccessKey": "your-secret" + } + }' + +# 获取当前存储信息 +curl http://localhost:3000/api/storage/storage/info +``` + +## 文件访问 + +### 统一下载接口 + +无论使用哪种存储类型,都通过统一的下载接口访问文件: + +```bash +# 统一下载接口(推荐) +GET http://localhost:3000/download/2024/01/01/abc123/example.jpg +``` + +### 本地存储 + +当使用本地存储时: + +- 下载接口会直接读取本地文件并返回 +- 支持内联显示(图片、PDF等)和下载 + +### S3 存储 + +当使用 S3 存储时: + +- 下载接口会重定向到 S3 URL +- 也可以直接访问 S3 URL(如果存储桶是公开的) + +```bash +# 直接访问 S3 URL +GET https://bucket.s3.region.amazonaws.com/2024/01/01/abc123/example.jpg +``` + +### 文件 URL 生成 + +```typescript +import { StorageUtils } from '@repo/storage'; + +const storageUtils = StorageUtils.getInstance(); + +// 生成下载 URL(推荐方式) +const fileUrl = storageUtils.generateFileUrl('file-id'); +// 结果: http://localhost:3000/download/file-id + +// 生成完整的公开访问 URL +const publicUrl = storageUtils.generateFileUrl('file-id', 'https://yourdomain.com'); +// 结果: https://yourdomain.com/download/file-id + +// 生成 S3 直接访问 URL(仅 S3 存储) +try { + const directUrl = storageUtils.generateDirectUrl('file-id'); + // 结果: https://bucket.s3.region.amazonaws.com/file-id +} catch (error) { + // 本地存储会抛出错误 +} +``` diff --git a/docs/STATIC_FILES.md b/docs/STATIC_FILES.md new file mode 100644 index 0000000..5b78f9d --- /dev/null +++ b/docs/STATIC_FILES.md @@ -0,0 +1,279 @@ +# 文件访问使用指南 + +本文档说明如何使用 `@repo/storage` 包提供的文件访问功能。 + +## 功能概述 + +存储包提供统一的文件访问接口: + +- **统一下载接口** (`/download/:fileId`) - 适用于所有存储类型,提供统一的文件访问 + +## 使用方法 + +### 1. 基础配置 + +```typescript +import { createStorageApp } from '@repo/storage'; + +// 创建包含所有功能的存储应用 +const storageApp = createStorageApp({ + apiBasePath: '/api/storage', // API 管理接口 + uploadPath: '/upload', // TUS 上传接口 + downloadPath: '/download', // 文件下载接口 +}); + +app.route('/', storageApp); +``` + +### 2. 分别配置功能 + +```typescript +import { createStorageRoutes, createTusUploadRoutes, createFileDownloadRoutes } from '@repo/storage'; + +const app = new Hono(); + +// 存储管理 API +app.route('/api/storage', createStorageRoutes()); + +// 文件上传 +app.route('/upload', createTusUploadRoutes()); + +// 文件下载(所有存储类型) +app.route('/download', createFileDownloadRoutes()); +``` + +## 文件访问方式 + +### 统一下载接口 + +无论使用哪种存储类型,都通过统一的下载接口访问文件: + +```bash +# 访问文件(支持内联显示和下载) +GET http://localhost:3000/download/2024/01/01/abc123/image.jpg +GET http://localhost:3000/download/2024/01/01/abc123/document.pdf +``` + +### 本地存储 + +当 `STORAGE_TYPE=local` 时: + +- 下载接口直接读取本地文件 +- 自动设置正确的 Content-Type +- 支持内联显示(`Content-Disposition: inline`) + +### S3 存储 + +当 `STORAGE_TYPE=s3` 时: + +- 下载接口重定向到 S3 URL +- 也可以直接访问 S3 URL(如果存储桶是公开的) + +```bash +# 直接访问 S3 URL(如果存储桶是公开的) +GET https://bucket.s3.region.amazonaws.com/2024/01/01/abc123/file.jpg +``` + +## 代码示例 + +### 生成文件访问 URL + +```typescript +import { StorageUtils } from '@repo/storage'; + +const storageUtils = StorageUtils.getInstance(); + +// 生成文件访问 URL +function getFileUrl(fileId: string) { + // 结果: http://localhost:3000/download/2024/01/01/abc123/file.jpg + return storageUtils.generateFileUrl(fileId); +} + +// 生成完整的公开访问 URL +function getPublicFileUrl(fileId: string) { + // 结果: https://yourdomain.com/download/2024/01/01/abc123/file.jpg + return storageUtils.generateFileUrl(fileId, 'https://yourdomain.com'); +} + +// 生成 S3 直接访问 URL(仅 S3 存储) +function getDirectUrl(fileId: string) { + try { + // S3 存储: https://bucket.s3.region.amazonaws.com/2024/01/01/abc123/file.jpg + return storageUtils.generateDirectUrl(fileId); + } catch (error) { + // 本地存储会抛出错误,使用下载接口 + return storageUtils.generateFileUrl(fileId); + } +} +``` + +### 在 React 组件中使用 + +```tsx +import { useState, useEffect } from 'react'; + +function FileDisplay({ fileId }: { fileId: string }) { + const [fileUrl, setFileUrl] = useState(''); + + useEffect(() => { + // 获取文件访问 URL + fetch(`/api/storage/resource/${fileId}`) + .then((res) => res.json()) + .then((data) => { + if (data.status === 'ready' && data.resource) { + // 生成文件访问 URL + const url = `/download/${fileId}`; + setFileUrl(url); + } + }); + }, [fileId]); + + if (!fileUrl) return
Loading...
; + + return ( +
+ {/* 图片会内联显示 */} + Uploaded file + + {/* 下载链接 */} + + 下载文件 + + + {/* PDF 等文档可以在新窗口打开 */} + + 在新窗口打开 + +
+ ); +} +``` + +### 文件类型处理 + +```typescript +function getFileDisplayUrl(fileId: string, mimeType: string) { + const baseUrl = `/download/${fileId}`; + + // 根据文件类型决定显示方式 + if (mimeType.startsWith('image/')) { + // 图片直接显示 + return baseUrl; + } else if (mimeType === 'application/pdf') { + // PDF 可以内联显示 + return baseUrl; + } else { + // 其他文件类型强制下载 + return `${baseUrl}?download=true`; + } +} +``` + +## 安全考虑 + +### 1. 访问控制 + +如需要权限验证,可以添加认证中间件: + +```typescript +import { createFileDownloadRoutes } from '@repo/storage'; + +const app = new Hono(); + +// 添加认证中间件 +app.use('/download/*', async (c, next) => { + // 检查用户权限 + const token = c.req.header('Authorization'); + if (!isValidToken(token)) { + return c.json({ error: 'Unauthorized' }, 401); + } + await next(); +}); + +// 添加文件下载服务 +app.route('/download', createFileDownloadRoutes()); +``` + +### 2. 文件类型限制 + +```typescript +app.use('/download/*', async (c, next) => { + const fileId = c.req.param('fileId'); + + // 从数据库获取文件信息 + const { resource } = await getResourceByFileId(fileId); + if (!resource) { + return c.json({ error: 'File not found' }, 404); + } + + // 检查文件类型 + const allowedTypes = ['image/jpeg', 'image/png', 'application/pdf']; + if (!allowedTypes.includes(resource.mimeType)) { + return c.json({ error: 'File type not allowed' }, 403); + } + + await next(); +}); +``` + +## 性能优化 + +### 1. 缓存设置 + +```typescript +app.use('/download/*', async (c, next) => { + await next(); + + // 设置缓存头 + c.header('Cache-Control', 'public, max-age=31536000'); // 1年 + c.header('ETag', generateETag(c.req.path)); +}); +``` + +### 2. CDN 配置 + +对于生产环境,建议使用 CDN: + +```typescript +import { StorageUtils } from '@repo/storage'; + +const storageUtils = StorageUtils.getInstance(); + +// 使用 CDN 域名 +const cdnUrl = 'https://cdn.yourdomain.com'; +const fileUrl = storageUtils.generateFileUrl(fileId, cdnUrl); +``` + +## 故障排除 + +### 常见问题 + +1. **404 文件未找到** + + - 检查文件是否存在于数据库 + - 确认文件路径是否正确 + - 检查文件权限(本地存储) + +2. **下载接口不工作** + + - 检查路由配置 + - 确认存储配置正确 + - 查看服务器日志 + +3. **S3 文件无法访问** + - 检查 S3 存储桶权限 + - 确认文件是否上传成功 + - 验证 S3 配置是否正确 + +### 调试方法 + +```bash +# 检查文件是否存在 +curl -I http://localhost:3000/download/2024/01/01/abc123/file.jpg + +# 检查存储配置 +curl http://localhost:3000/api/storage/storage/info + +# 检查文件信息 +curl http://localhost:3000/api/storage/resource/2024/01/01/abc123/file.jpg +``` diff --git a/env.example b/env.example new file mode 100644 index 0000000..76e441e --- /dev/null +++ b/env.example @@ -0,0 +1,71 @@ +# =========================================== +# 存储配置 (@repo/storage) +# =========================================== + +# 存储类型: local | s3 +STORAGE_TYPE=local + +# 上传文件过期时间(毫秒),0表示不过期 +UPLOAD_EXPIRATION_MS=0 + +# =========================================== +# 本地存储配置 (当 STORAGE_TYPE=local 时) +# =========================================== + +# 本地存储目录路径 +UPLOAD_DIR=./uploads + +# =========================================== +# S3 存储配置 (当 STORAGE_TYPE=s3 时) +# =========================================== + +# S3 存储桶名称 (必需) +S3_BUCKET= + +# S3 区域 (必需) +S3_REGION=us-east-1 + +# S3 访问密钥 ID (必需) +S3_ACCESS_KEY_ID= + +# S3 访问密钥 (必需) +S3_SECRET_ACCESS_KEY= + +# 自定义 S3 端点 (可选,用于 MinIO、阿里云 OSS 等) +S3_ENDPOINT= + +# 是否强制使用路径样式 (可选) +S3_FORCE_PATH_STYLE=false + +# 分片上传大小,单位字节 (可选,默认 8MB) +S3_PART_SIZE=8388608 + +# 最大并发上传数 (可选) +S3_MAX_CONCURRENT_UPLOADS=60 + +# =========================================== +# 数据库配置 +# =========================================== + +# 数据库连接字符串 +DATABASE_URL="postgresql://username:password@localhost:5432/database" + +# =========================================== +# Redis 配置 +# =========================================== + +# Redis 连接字符串 +REDIS_URL="redis://localhost:6379" + +# =========================================== +# 应用配置 +# =========================================== + +# 应用端口 +PORT=3000 + +# 应用环境 +NODE_ENV=development + +# CORS 允许的源 +CORS_ORIGIN=http://localhost:3001 \ No newline at end of file diff --git a/packages/storage/.env.example b/packages/storage/.env.example new file mode 100644 index 0000000..72f65dc --- /dev/null +++ b/packages/storage/.env.example @@ -0,0 +1,3 @@ +STORAGE_TYPE=local +UPLOAD_DIR=/opt/projects/nice/uploads +UPLOAD_EXPIRATION_MS=0 \ No newline at end of file diff --git a/packages/storage/README.md b/packages/storage/README.md new file mode 100644 index 0000000..c5e4fc3 --- /dev/null +++ b/packages/storage/README.md @@ -0,0 +1,324 @@ +# @repo/storage + +一个完全兼容 Hono 的存储解决方案,支持本地存储和 S3 兼容存储,提供 TUS 协议上传、文件管理和 REST API。 + +## 特性 + +- 🚀 **多存储支持**: 支持本地文件系统和 S3 兼容存储 +- 📤 **TUS 协议**: 支持可恢复的文件上传 +- 🔧 **Hono 集成**: 提供开箱即用的 Hono 中间件 +- 📊 **文件管理**: 完整的文件生命周期管理 +- 🗄️ **数据库集成**: 与 Prisma 深度集成 +- ⏰ **自动清理**: 支持过期文件自动清理 +- 🔄 **存储迁移**: 支持不同存储类型间的数据迁移 + +## 安装 + +```bash +npm install @repo/storage +``` + +## 环境变量配置 + +### 基础配置 + +| 变量名 | 类型 | 默认值 | 描述 | +| ---------------------- | --------------- | ------- | ------------------------------------- | +| `STORAGE_TYPE` | `local` \| `s3` | `local` | 存储类型选择 | +| `UPLOAD_EXPIRATION_MS` | `number` | `0` | 上传文件过期时间(毫秒),0表示不过期 | + +### 本地存储配置 + +当 `STORAGE_TYPE=local` 时需要配置: + +| 变量名 | 类型 | 默认值 | 描述 | +| ------------ | -------- | ----------- | ---------------- | +| `UPLOAD_DIR` | `string` | `./uploads` | 本地存储目录路径 | + +### S3 存储配置 + +当 `STORAGE_TYPE=s3` 时需要配置: + +| 变量名 | 类型 | 默认值 | 描述 | 必需 | +| --------------------------- | --------- | ----------- | ---------------------------------- | ---- | +| `S3_BUCKET` | `string` | - | S3 存储桶名称 | ✅ | +| `S3_REGION` | `string` | `us-east-1` | S3 区域 | ✅ | +| `S3_ACCESS_KEY_ID` | `string` | - | S3 访问密钥 ID | ✅ | +| `S3_SECRET_ACCESS_KEY` | `string` | - | S3 访问密钥 | ✅ | +| `S3_ENDPOINT` | `string` | - | 自定义 S3 端点(用于兼容其他服务) | ❌ | +| `S3_FORCE_PATH_STYLE` | `boolean` | `false` | 是否强制使用路径样式 | ❌ | +| `S3_PART_SIZE` | `number` | `8388608` | 分片上传大小(8MB) | ❌ | +| `S3_MAX_CONCURRENT_UPLOADS` | `number` | `60` | 最大并发上传数 | ❌ | + +## 配置示例 + +### 本地存储配置 + +```bash +# .env +STORAGE_TYPE=local +UPLOAD_DIR=./uploads +UPLOAD_EXPIRATION_MS=86400000 # 24小时过期 +``` + +### AWS S3 配置 + +```bash +# .env +STORAGE_TYPE=s3 +S3_BUCKET=my-app-uploads +S3_REGION=us-west-2 +S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE +S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +UPLOAD_EXPIRATION_MS=604800000 # 7天过期 +``` + +### MinIO 配置 + +```bash +# .env +STORAGE_TYPE=s3 +S3_BUCKET=uploads +S3_REGION=us-east-1 +S3_ACCESS_KEY_ID=minioadmin +S3_SECRET_ACCESS_KEY=minioadmin +S3_ENDPOINT=http://localhost:9000 +S3_FORCE_PATH_STYLE=true +``` + +### 阿里云 OSS 配置 + +```bash +# .env +STORAGE_TYPE=s3 +S3_BUCKET=my-oss-bucket +S3_REGION=oss-cn-hangzhou +S3_ACCESS_KEY_ID=your-access-key-id +S3_SECRET_ACCESS_KEY=your-access-key-secret +S3_ENDPOINT=https://oss-cn-hangzhou.aliyuncs.com +S3_FORCE_PATH_STYLE=false +``` + +### 腾讯云 COS 配置 + +```bash +# .env +STORAGE_TYPE=s3 +S3_BUCKET=my-cos-bucket-1234567890 +S3_REGION=ap-beijing +S3_ACCESS_KEY_ID=your-secret-id +S3_SECRET_ACCESS_KEY=your-secret-key +S3_ENDPOINT=https://cos.ap-beijing.myqcloud.com +S3_FORCE_PATH_STYLE=false +``` + +## 快速开始 + +### 1. 基础使用 + +```typescript +import { createStorageApp, startCleanupScheduler } from '@repo/storage'; +import { Hono } from 'hono'; + +const app = new Hono(); + +// 创建存储应用 +const storageApp = createStorageApp({ + apiBasePath: '/api/storage', // API 路径 + uploadPath: '/upload', // 上传路径 +}); + +// 挂载存储应用 +app.route('/', storageApp); + +// 启动清理调度器 +startCleanupScheduler(); +``` + +### 2. 分别使用 API 和上传功能 + +```typescript +import { createStorageRoutes, createTusUploadRoutes } from '@repo/storage'; + +const app = new Hono(); + +// 只添加存储管理 API +app.route('/api/storage', createStorageRoutes()); + +// 只添加文件上传功能 +app.route('/upload', createTusUploadRoutes()); +``` + +### 3. 使用存储管理器 + +```typescript +import { StorageManager, StorageUtils } from '@repo/storage'; + +// 获取存储管理器实例 +const storageManager = StorageManager.getInstance(); + +// 获取存储信息 +const storageInfo = storageManager.getStorageInfo(); +console.log('当前存储类型:', storageInfo.type); + +// 使用存储工具 +const storageUtils = StorageUtils.getInstance(); + +// 生成文件访问 URL(统一使用下载接口) +const fileUrl = storageUtils.generateFileUrl('2024/01/01/abc123/file.jpg'); +// 结果: http://localhost:3000/download/2024/01/01/abc123/file.jpg + +// 生成完整的公开访问 URL +const publicUrl = storageUtils.generateFileUrl('2024/01/01/abc123/file.jpg', 'https://yourdomain.com'); +// 结果: https://yourdomain.com/download/2024/01/01/abc123/file.jpg + +// 生成 S3 直接访问 URL(仅 S3 存储) +try { + const directUrl = storageUtils.generateDirectUrl('2024/01/01/abc123/file.jpg'); + // S3 存储: https://bucket.s3.region.amazonaws.com/2024/01/01/abc123/file.jpg +} catch (error) { + // 本地存储会抛出错误 +} + +// 检查文件是否存在 +const exists = await storageUtils.fileExists('file-id'); +``` + +### 4. 分别配置不同功能 + +```typescript +import { createStorageRoutes, createTusUploadRoutes, createFileDownloadRoutes } from '@repo/storage'; + +const app = new Hono(); + +// 只添加存储管理 API +app.route('/api/storage', createStorageRoutes()); + +// 只添加文件上传功能 +app.route('/upload', createTusUploadRoutes()); + +// 只添加文件下载功能(所有存储类型) +app.route('/download', createFileDownloadRoutes()); +``` + +## API 端点 + +### 文件资源管理 + +- `GET /api/storage/resource/:fileId` - 获取文件资源信息 +- `GET /api/storage/resources` - 获取所有资源 +- `GET /api/storage/resources/storage/:storageType` - 按存储类型获取资源 +- `GET /api/storage/resources/status/:status` - 按状态获取资源 +- `GET /api/storage/resources/uploading` - 获取正在上传的资源 +- `DELETE /api/storage/resource/:id` - 删除资源 +- `PATCH /api/storage/resource/:id` - 更新资源 + +### 文件访问和下载 + +- `GET /download/:fileId` - 文件下载和访问(支持所有存储类型) + +### 统计和管理 + +- `GET /api/storage/stats` - 获取资源统计信息 +- `POST /api/storage/cleanup` - 手动清理过期上传 +- `POST /api/storage/cleanup/by-status` - 按状态清理资源 +- `POST /api/storage/migrate-storage` - 迁移存储类型 + +### 存储配置 + +- `GET /api/storage/storage/info` - 获取存储信息 +- `POST /api/storage/storage/switch` - 切换存储配置 +- `POST /api/storage/storage/validate` - 验证存储配置 + +### 文件上传 + +- `POST /upload` - TUS 协议文件上传 +- `PATCH /upload/:id` - 续传文件 +- `HEAD /upload/:id` - 获取上传状态 + +## 数据库操作 + +```typescript +import { + getAllResources, + getResourceByFileId, + createResource, + updateResourceStatus, + deleteResource, +} from '@repo/storage'; + +// 获取所有资源 +const resources = await getAllResources(); + +// 根据文件ID获取资源 +const { status, resource } = await getResourceByFileId('file-id'); + +// 创建新资源 +const newResource = await createResource({ + fileId: 'unique-file-id', + filename: 'example.jpg', + size: 1024000, + mimeType: 'image/jpeg', + storageType: 'local', +}); +``` + +## 文件生命周期 + +1. **上传开始**: 创建资源记录,状态为 `UPLOADING` +2. **上传完成**: 状态更新为 `UPLOADED` +3. **处理中**: 状态可更新为 `PROCESSING` +4. **处理完成**: 状态更新为 `PROCESSED` +5. **清理**: 过期文件自动清理 + +## 存储迁移 + +支持在不同存储类型之间迁移数据: + +```bash +# API 调用示例 +curl -X POST http://localhost:3000/api/storage/migrate-storage \ + -H "Content-Type: application/json" \ + -d '{"from": "local", "to": "s3"}' +``` + +## 安全考虑 + +1. **环境变量**: 敏感信息(如 S3 密钥)应存储在环境变量中 +2. **访问控制**: 建议在生产环境中添加适当的身份验证 +3. **CORS 配置**: 根据需要配置跨域访问策略 +4. **文件验证**: 建议添加文件类型和大小验证 + +## 故障排除 + +### 常见问题 + +1. **找不到模块错误**: 确保已正确安装依赖包 +2. **S3 连接失败**: 检查网络连接和凭据配置 +3. **本地存储权限**: 确保应用有写入本地目录的权限 +4. **上传失败**: 检查文件大小限制和存储空间 + +### 调试模式 + +启用详细日志: + +```bash +DEBUG=storage:* npm start +``` + +## 许可证 + +MIT License + +## 贡献 + +欢迎提交 Issue 和 Pull Request! + +## 更新日志 + +### v2.0.0 + +- 重构为模块化架构 +- 添加完整的 TypeScript 支持 +- 支持多种 S3 兼容服务 +- 改进的错误处理和日志记录 diff --git a/packages/storage/docs/S3_DOWNLOAD_MECHANISM.md b/packages/storage/docs/S3_DOWNLOAD_MECHANISM.md new file mode 100644 index 0000000..ec61e88 --- /dev/null +++ b/packages/storage/docs/S3_DOWNLOAD_MECHANISM.md @@ -0,0 +1,110 @@ +# S3存储下载机制说明 + +## 问题背景 + +在文件上传系统中,我们使用了两种存储类型: + +- **本地存储(Local)**:文件存储在服务器本地文件系统 +- **S3存储(S3)**:文件存储在AWS S3或兼容的对象存储服务中 + +对于文件访问,我们使用了目录格式的 `fileId`,例如:`2025/05/28/RHwt8AkkZp` + +## 存储结构差异 + +### 本地存储 + +- **fileId**:`2025/05/28/RHwt8AkkZp` (目录路径) +- **实际存储**:`/uploads/2025/05/28/RHwt8AkkZp/filename.ext` +- **下载方式**:扫描目录,找到实际文件,返回文件流 + +### S3存储 + +- **fileId**:`2025/05/28/RHwt8AkkZp` (目录路径) +- **S3 Key**:`2025/05/28/RHwt8AkkZp/filename.ext` (完整对象路径) +- **下载方式**:重定向到S3 URL + +## 核心问题 + +S3存储中,对象的完整路径(S3 Key)包含文件名,但我们的 `fileId` 只是目录路径,缺少文件名部分。 + +## 解决方案 + +### 1. 文件名重建策略 + +我们通过以下方式重建完整的S3路径: + +```typescript +const fileName = resource.title || 'file'; +const fullS3Key = `${fileId}/${fileName}`; +``` + +### 2. URL生成逻辑 + +```typescript +// AWS S3 +const s3Url = `https://${bucket}.s3.${region}.amazonaws.com/${fullS3Key}`; + +// 自定义S3兼容服务(如MinIO) +const s3Url = `${endpoint}/${bucket}/${fullS3Key}`; +``` + +### 3. 下载流程 + +1. 从数据库获取文件信息(fileId + resource.title) +2. 重建完整S3 Key:`${fileId}/${fileName}` +3. 生成S3直接访问URL +4. 302重定向到S3 URL,让客户端直接从S3下载 + +## 优势 + +### 性能优势 + +- **302重定向**:避免服务器中转,减少带宽消耗 +- **直接下载**:客户端直接从S3下载,速度更快 +- **CDN友好**:可配合CloudFront等CDN使用 + +### 安全考虑 + +- **公开读取**:需要确保S3 bucket配置了适当的公开读取权限 +- **预签名URL**:未来可扩展支持预签名URL用于私有文件 + +## 局限性 + +### 文件名依赖 + +- 依赖数据库中存储的 `resource.title` 字段 +- 如果文件名不匹配,会导致404错误 + +### 替代方案 + +如果需要更可靠的方案,可以考虑: + +1. **存储完整S3 Key**:在数据库中存储完整的S3对象路径 +2. **S3 ListObjects API**:动态查询S3中的实际对象(会增加API调用成本) + +## 环境配置 + +确保S3配置正确: + +```env +STORAGE_TYPE=s3 +S3_BUCKET=your-bucket-name +S3_REGION=us-east-1 +S3_ACCESS_KEY_ID=your-access-key +S3_SECRET_ACCESS_KEY=your-secret-key +S3_ENDPOINT=https://s3.amazonaws.com # 可选,用于其他S3兼容服务 +``` + +## 测试验证 + +使用以下URL格式测试下载: + +``` +/download/2025%2F05%2F28%2FRHwt8AkkZp +``` + +应该会302重定向到: + +``` +https://your-bucket.s3.us-east-1.amazonaws.com/2025/05/28/RHwt8AkkZp/filename.ext +``` diff --git a/packages/storage/package.json b/packages/storage/package.json new file mode 100644 index 0000000..aa68897 --- /dev/null +++ b/packages/storage/package.json @@ -0,0 +1,62 @@ +{ + "name": "@repo/storage", + "version": "2.0.0", + "description": "Storage implementation for Hono - 完全兼容 Hono 的 Storage", + "main": "dist/index.js", + "types": "dist/index.d.ts", + "scripts": { + "build": "tsc", + "dev": "tsc --watch", + "clean": "rm -rf dist" + }, + "dependencies": { + "@hono/zod-validator": "^0.5.0", + "@repo/db": "workspace:*", + "@repo/tus": "workspace:*", + "hono": "^4.7.10", + "ioredis": "5.4.1", + "jose": "^6.0.11", + "nanoid": "^5.1.5", + "transliteration": "^2.3.5", + "zod": "^3.25.23" + }, + "devDependencies": { + "@types/node": "^22.15.21", + "typescript": "^5.0.0" + }, + "peerDependencies": { + "@repo/db": "workspace:*", + "@repo/tus": "workspace:*", + "hono": "^4.0.0", + "ioredis": "^5.0.0" + }, + "exports": { + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.js", + "require": "./dist/index.js" + } + }, + "files": [ + "dist", + "README.md" + ], + "keywords": [ + "storage", + "hono", + "tus", + "upload", + "typescript" + ], + "author": "Your Name", + "license": "MIT", + "repository": { + "type": "git", + "url": "https://github.com/your-org/your-repo.git", + "directory": "packages/storage" + }, + "bugs": { + "url": "https://github.com/your-org/your-repo/issues" + }, + "homepage": "https://github.com/your-org/your-repo/tree/main/packages/storage#readme" +} diff --git a/apps/backend/src/upload/storage.adapter.ts b/packages/storage/src/core/adapter.ts similarity index 89% rename from apps/backend/src/upload/storage.adapter.ts rename to packages/storage/src/core/adapter.ts index 6ba69fd..4104e22 100644 --- a/apps/backend/src/upload/storage.adapter.ts +++ b/packages/storage/src/core/adapter.ts @@ -1,33 +1,6 @@ import { FileStore, S3Store } from '@repo/tus'; import type { DataStore } from '@repo/tus'; - -// 存储类型枚举 -export enum StorageType { - LOCAL = 'local', - S3 = 's3', -} - -// 存储配置接口 -export interface StorageConfig { - type: StorageType; - // 本地存储配置 - local?: { - directory: string; - expirationPeriodInMilliseconds?: number; - }; - // S3 存储配置 - s3?: { - bucket: string; - region: string; - accessKeyId: string; - secretAccessKey: string; - endpoint?: string; // 用于兼容其他 S3 兼容服务 - forcePathStyle?: boolean; - partSize?: number; - maxConcurrentPartUploads?: number; - expirationPeriodInMilliseconds?: number; - }; -} +import { StorageType, StorageConfig } from '../types'; // 从环境变量获取存储配置 export function getStorageConfig(): StorageConfig { diff --git a/packages/storage/src/core/index.ts b/packages/storage/src/core/index.ts new file mode 100644 index 0000000..095283d --- /dev/null +++ b/packages/storage/src/core/index.ts @@ -0,0 +1,5 @@ +// 存储适配器 +export * from './adapter'; + +// 便捷导出 +export { StorageManager } from './adapter'; diff --git a/packages/storage/src/database/index.ts b/packages/storage/src/database/index.ts new file mode 100644 index 0000000..4811dbb --- /dev/null +++ b/packages/storage/src/database/index.ts @@ -0,0 +1,2 @@ +// 数据库操作 +export * from './operations'; diff --git a/apps/backend/src/upload/upload.index.ts b/packages/storage/src/database/operations.ts similarity index 75% rename from apps/backend/src/upload/upload.index.ts rename to packages/storage/src/database/operations.ts index 83066fc..3cbe6e9 100644 --- a/apps/backend/src/upload/upload.index.ts +++ b/packages/storage/src/database/operations.ts @@ -1,6 +1,6 @@ import { prisma } from '@repo/db'; import type { Resource } from '@repo/db'; -import { StorageType } from './storage.adapter'; +import { StorageType } from '../types'; export async function getResourceByFileId(fileId: string): Promise<{ status: string; resource?: Resource }> { const resource = await prisma.resource.findFirst({ @@ -11,7 +11,10 @@ export async function getResourceByFileId(fileId: string): Promise<{ status: str return { status: 'pending' }; } - return { status: 'ready', resource }; + return { + status: resource.status || 'unknown', + resource, + }; } export async function getAllResources(): Promise { @@ -114,3 +117,37 @@ export async function migrateResourcesStorageType( return { count: result.count }; } + +export async function createResource(data: { + fileId: string; + filename: string; + size: number; + mimeType?: string | null; + storageType: StorageType; + status?: string; + hash?: string; +}): Promise { + return prisma.resource.create({ + data: { + fileId: data.fileId, + title: data.filename, + type: data.mimeType, + storageType: data.storageType, + status: data.status || 'UPLOADING', + meta: { + size: data.size, + hash: data.hash, + }, + }, + }); +} + +export async function updateResourceStatus(fileId: string, status: string, additionalData?: any): Promise { + return prisma.resource.update({ + where: { fileId }, + data: { + status, + ...additionalData, + }, + }); +} diff --git a/packages/storage/src/index.ts b/packages/storage/src/index.ts new file mode 100644 index 0000000..ea4272a --- /dev/null +++ b/packages/storage/src/index.ts @@ -0,0 +1,21 @@ +// 类型定义 +export * from './types'; + +// 核心功能 +export * from './core'; + +// 数据库操作 +export * from './database'; + +// 服务层 +export * from './services'; + +// Hono 中间件 +export * from './middleware'; + +// 便捷的默认导出 +export { StorageManager } from './core'; +export { StorageUtils } from './services'; +export { getTusServer, handleTusRequest } from './services'; +export { startCleanupScheduler, triggerCleanup } from './services'; +export { createStorageApp, createStorageRoutes, createTusUploadRoutes, createFileDownloadRoutes } from './middleware'; diff --git a/packages/storage/src/middleware/hono.ts b/packages/storage/src/middleware/hono.ts new file mode 100644 index 0000000..34ef29b --- /dev/null +++ b/packages/storage/src/middleware/hono.ts @@ -0,0 +1,511 @@ +import { Hono } from 'hono'; +import { handleTusRequest, cleanupExpiredUploads, getStorageInfo } from '../services/tus'; +import { + getResourceByFileId, + getAllResources, + deleteResource, + updateResource, + getResourcesByStorageType, + getResourcesByStatus, + getUploadingResources, + getResourceStats, + migrateResourcesStorageType, +} from '../database/operations'; +import { StorageManager, validateStorageConfig } from '../core/adapter'; +import { StorageType, type StorageConfig } from '../types'; +import { prisma } from '@repo/db'; + +/** + * 创建存储相关的 Hono 路由 + * @param basePath 基础路径,默认为 '/api/storage' + * @returns Hono 应用实例 + */ +export function createStorageRoutes(basePath: string = '/api/storage') { + const app = new Hono(); + + // 获取文件资源信息 + app.get('/resource/:fileId', async (c) => { + const encodedFileId = c.req.param('fileId'); + const fileId = decodeURIComponent(encodedFileId); + console.log('API request - Encoded fileId:', encodedFileId); + console.log('API request - Decoded fileId:', fileId); + const result = await getResourceByFileId(fileId); + return c.json(result); + }); + + // 获取所有资源 + app.get('/resources', async (c) => { + const resources = await getAllResources(); + return c.json(resources); + }); + + // 根据存储类型获取资源 + app.get('/resources/storage/:storageType', async (c) => { + const storageType = c.req.param('storageType') as StorageType; + const resources = await getResourcesByStorageType(storageType); + return c.json(resources); + }); + + // 根据状态获取资源 + app.get('/resources/status/:status', async (c) => { + const status = c.req.param('status'); + const resources = await getResourcesByStatus(status); + return c.json(resources); + }); + + // 获取正在上传的资源 + app.get('/resources/uploading', async (c) => { + const resources = await getUploadingResources(); + return c.json(resources); + }); + + // 获取资源统计信息 + app.get('/stats', async (c) => { + const stats = await getResourceStats(); + return c.json(stats); + }); + + // 删除资源 + app.delete('/resource/:id', async (c) => { + const id = c.req.param('id'); + const result = await deleteResource(id); + return c.json(result); + }); + + // 更新资源 + app.patch('/resource/:id', async (c) => { + const id = c.req.param('id'); + const data = await c.req.json(); + const result = await updateResource(id, data); + return c.json(result); + }); + + // 迁移资源存储类型(批量更新数据库中的存储类型标记) + app.post('/migrate-storage', async (c) => { + try { + const { from, to } = await c.req.json(); + const result = await migrateResourcesStorageType(from as StorageType, to as StorageType); + return c.json({ + success: true, + message: `Migrated ${result.count} resources from ${from} to ${to}`, + count: result.count, + }); + } catch (error) { + console.error('Failed to migrate storage type:', error); + return c.json( + { + success: false, + error: error instanceof Error ? error.message : 'Unknown error', + }, + 400, + ); + } + }); + + // 清理过期上传 + app.post('/cleanup', async (c) => { + const result = await cleanupExpiredUploads(); + return c.json(result); + }); + + // 手动清理指定状态的资源 + app.post('/cleanup/by-status', async (c) => { + try { + const { status, olderThanDays } = await c.req.json(); + const cutoffDate = new Date(); + cutoffDate.setDate(cutoffDate.getDate() - (olderThanDays || 30)); + + const deletedResources = await prisma.resource.deleteMany({ + where: { + status, + createdAt: { + lt: cutoffDate, + }, + }, + }); + + return c.json({ + success: true, + message: `Deleted ${deletedResources.count} resources with status ${status}`, + count: deletedResources.count, + }); + } catch (error) { + console.error('Failed to cleanup by status:', error); + return c.json( + { + success: false, + error: error instanceof Error ? error.message : 'Unknown error', + }, + 400, + ); + } + }); + + // 获取存储信息 + app.get('/storage/info', async (c) => { + const storageInfo = getStorageInfo(); + return c.json(storageInfo); + }); + + // 切换存储类型(需要重启应用) + app.post('/storage/switch', async (c) => { + try { + const newConfig = (await c.req.json()) as StorageConfig; + const storageManager = StorageManager.getInstance(); + await storageManager.switchStorage(newConfig); + + return c.json({ + success: true, + message: 'Storage configuration updated. Please restart the application for changes to take effect.', + newType: newConfig.type, + }); + } catch (error) { + console.error('Failed to switch storage:', error); + return c.json( + { + success: false, + error: error instanceof Error ? error.message : 'Unknown error', + }, + 400, + ); + } + }); + + // 验证存储配置 + app.post('/storage/validate', async (c) => { + try { + const config = (await c.req.json()) as StorageConfig; + const errors = validateStorageConfig(config); + + if (errors.length > 0) { + return c.json({ valid: false, errors }, 400); + } + + return c.json({ valid: true, message: 'Storage configuration is valid' }); + } catch (error) { + return c.json( + { + valid: false, + errors: [error instanceof Error ? error.message : 'Invalid JSON'], + }, + 400, + ); + } + }); + + return app; +} + +/** + * 创建TUS上传处理路由 + * @param uploadPath 上传路径,默认为 '/upload' + * @returns Hono 应用实例 + */ +export function createTusUploadRoutes(uploadPath: string = '/upload') { + const app = new Hono(); + + // TUS 协议处理 - 使用通用处理器 + app.all('/*', async (c) => { + try { + // 创建适配的请求和响应对象 + const adaptedReq = createNodeRequestAdapter(c); + const adaptedRes = createNodeResponseAdapter(c); + + await handleTusRequest(adaptedReq, adaptedRes); + return adaptedRes.getResponse(); + } catch (error) { + console.error('TUS request error:', error); + return c.json({ error: 'Upload request failed' }, 500); + } + }); + + return app; +} + +// Node.js 请求适配器 +function createNodeRequestAdapter(c: any) { + const honoReq = c.req; + const url = new URL(honoReq.url); + + // 导入Node.js模块 + const { Readable } = require('stream'); + const { EventEmitter } = require('events'); + + // 创建一个继承自Readable的适配器类 + class TusRequestAdapter extends Readable { + method: string; + url: string; + headers: Record; + httpVersion: string; + complete: boolean; + private reader: ReadableStreamDefaultReader | null = null; + private _reading: boolean = false; + + constructor() { + super(); + this.method = honoReq.method; + this.url = url.pathname + url.search; + this.headers = honoReq.header() || {}; + this.httpVersion = '1.1'; + this.complete = false; + + // 如果有请求体,获取reader + if (honoReq.method !== 'GET' && honoReq.method !== 'HEAD' && honoReq.raw.body) { + this.reader = honoReq.raw.body.getReader(); + } + } + + _read() { + if (this._reading || !this.reader) { + return; + } + + this._reading = true; + + this.reader + .read() + .then(({ done, value }) => { + this._reading = false; + if (done) { + this.push(null); // 结束流 + this.complete = true; + } else { + // 确保我们推送的是正确的二进制数据 + const buffer = Buffer.from(value); + this.push(buffer); + } + }) + .catch((error) => { + this._reading = false; + this.destroy(error); + }); + } + + // 模拟IncomingMessage的destroy方法 + destroy(error?: Error) { + if (this.reader) { + this.reader.cancel().catch(() => { + // 忽略取消错误 + }); + this.reader = null; + } + super.destroy(error); + } + } + + return new TusRequestAdapter(); +} + +// Node.js 响应适配器 +function createNodeResponseAdapter(c: any) { + let statusCode = 200; + let headers: Record = {}; + let body: any = null; + + const adapter = { + statusCode, + setHeader: (name: string, value: string) => { + headers[name] = value; + }, + getHeader: (name: string) => { + return headers[name]; + }, + writeHead: (code: number, reasonOrHeaders?: any, headersObj?: any) => { + statusCode = code; + if (typeof reasonOrHeaders === 'object') { + Object.assign(headers, reasonOrHeaders); + } + if (headersObj) { + Object.assign(headers, headersObj); + } + }, + write: (chunk: any) => { + if (body === null) { + body = chunk; + } else if (typeof body === 'string' && typeof chunk === 'string') { + body += chunk; + } else { + // 处理 Buffer 或其他类型 + body = chunk; + } + }, + end: (data?: any) => { + if (data !== undefined) { + body = data; + } + }, + // 添加事件方法 + on: (event: string, handler: Function) => { + // 简单的空实现 + }, + emit: (event: string, ...args: any[]) => { + // 简单的空实现 + }, + // 获取最终的 Response 对象 + getResponse: () => { + if (body === null || body === undefined) { + return new Response(null, { + status: statusCode, + headers: headers, + }); + } + + return new Response(body, { + status: statusCode, + headers: headers, + }); + }, + }; + + return adapter; +} + +/** + * 创建文件下载路由(支持所有存储类型) + * @param downloadPath 下载路径,默认为 '/download' + * @returns Hono 应用实例 + */ +export function createFileDownloadRoutes(downloadPath: string = '/download') { + const app = new Hono(); + + // 通过文件ID下载文件 + app.get('/:fileId', async (c) => { + try { + // 获取并解码fileId + const encodedFileId = c.req.param('fileId'); + const fileId = decodeURIComponent(encodedFileId); + + console.log('Download request - Encoded fileId:', encodedFileId); + console.log('Download request - Decoded fileId:', fileId); + + const storageManager = StorageManager.getInstance(); + const storageType = storageManager.getStorageType(); + + // 从数据库获取文件信息 + const { status, resource } = await getResourceByFileId(fileId); + if (status !== 'UPLOADED' || !resource) { + return c.json({ error: `File not found or not ready. Status: ${status}, FileId: ${fileId}` }, 404); + } + + if (storageType === StorageType.LOCAL) { + // 本地存储:直接读取文件 + const config = storageManager.getStorageConfig(); + const uploadDir = config.local?.directory || './uploads'; + + // fileId 是目录路径格式,直接使用 + const fileDir = `${uploadDir}/${fileId}`; + + try { + // 使用 Node.js fs 而不是 Bun.file + const fs = await import('fs'); + const path = await import('path'); + + // 检查目录是否存在 + if (!fs.existsSync(fileDir)) { + return c.json({ error: `File directory not found: ${fileDir}` }, 404); + } + + // 读取目录内容,找到实际的文件(排除 .json 文件) + const files = fs.readdirSync(fileDir).filter((f) => !f.endsWith('.json')); + if (files.length === 0) { + return c.json({ error: `No file found in directory: ${fileDir}` }, 404); + } + + // 通常只有一个文件,取第一个 + const actualFileName = files[0]; + if (!actualFileName) { + return c.json({ error: 'No valid file found' }, 404); + } + const filePath = path.join(fileDir, actualFileName); + + // 获取文件统计信息 + const stats = fs.statSync(filePath); + const fileSize = stats.size; + + // 设置响应头 + c.header('Content-Type', resource.type || 'application/octet-stream'); + c.header('Content-Length', fileSize.toString()); + c.header('Content-Disposition', `inline; filename="${actualFileName}"`); + + // 返回文件流 + const fileStream = fs.createReadStream(filePath); + return new Response(fileStream as any); + } catch (error) { + console.error('Error reading local file:', error); + return c.json({ error: 'Failed to read file' }, 500); + } + } else if (storageType === StorageType.S3) { + // S3 存储:通过已配置的dataStore获取文件信息 + const dataStore = storageManager.getDataStore(); + + try { + // 对于S3存储,我们需要根据fileId构建完整路径 + // 由于S3Store的client是私有的,我们先尝试通过getUpload来验证文件存在 + await (dataStore as any).getUpload(fileId + '/dummy'); // 这会失败,但能验证连接 + } catch (error: any) { + // 如果是FILE_NOT_FOUND以外的错误,说明连接有问题 + if (error.message && !error.message.includes('FILE_NOT_FOUND')) { + console.error('S3 connection error:', error); + return c.json({ error: 'Failed to access S3 storage' }, 500); + } + } + + // 构建S3 URL - 使用resource信息重建完整路径 + // 这里我们假设文件名就是resource.title + const config = storageManager.getStorageConfig(); + const s3Config = config.s3!; + const fileName = resource.title || 'file'; + const fullS3Key = `${fileId}/${fileName}`; + + // 生成 S3 URL + let s3Url: string; + if (s3Config.endpoint && s3Config.endpoint !== 'https://s3.amazonaws.com') { + // 自定义 S3 兼容服务 + s3Url = `${s3Config.endpoint}/${s3Config.bucket}/${fullS3Key}`; + } else { + // AWS S3 + s3Url = `https://${s3Config.bucket}.s3.${s3Config.region}.amazonaws.com/${fullS3Key}`; + } + + console.log(`Redirecting to S3 URL: ${s3Url}`); + // 重定向到 S3 URL + return c.redirect(s3Url, 302); + } + + return c.json({ error: 'Unsupported storage type' }, 500); + } catch (error) { + console.error('Download error:', error); + return c.json({ error: 'Internal server error' }, 500); + } + }); + + return app; +} + +/** + * 创建完整的存储应用,包含API和上传功能 + * @param options 配置选项 + * @returns Hono 应用实例 + */ +export function createStorageApp( + options: { + apiBasePath?: string; + uploadPath?: string; + downloadPath?: string; + } = {}, +) { + const { apiBasePath = '/api/storage', uploadPath = '/upload', downloadPath = '/download' } = options; + + const app = new Hono(); + + // 添加存储API路由 + app.route(apiBasePath, createStorageRoutes()); + + // 添加TUS上传路由 + app.route(uploadPath, createTusUploadRoutes()); + + // 添加文件下载路由 + app.route(downloadPath, createFileDownloadRoutes()); + + return app; +} diff --git a/packages/storage/src/middleware/index.ts b/packages/storage/src/middleware/index.ts new file mode 100644 index 0000000..23a3b24 --- /dev/null +++ b/packages/storage/src/middleware/index.ts @@ -0,0 +1,5 @@ +// Hono 中间件 +export * from './hono'; + +// 便捷导出 +export { createStorageApp, createStorageRoutes, createTusUploadRoutes, createFileDownloadRoutes } from './hono'; diff --git a/packages/storage/src/services/index.ts b/packages/storage/src/services/index.ts new file mode 100644 index 0000000..4b689b1 --- /dev/null +++ b/packages/storage/src/services/index.ts @@ -0,0 +1,13 @@ +// TUS 上传处理 +export * from './tus'; + +// 存储工具 +export * from './utils'; + +// 调度器 +export * from './scheduler'; + +// 便捷导出 +export { StorageUtils } from './utils'; +export { getTusServer, handleTusRequest } from './tus'; +export { startCleanupScheduler, triggerCleanup } from './scheduler'; diff --git a/apps/backend/src/upload/scheduler.ts b/packages/storage/src/services/scheduler.ts similarity index 100% rename from apps/backend/src/upload/scheduler.ts rename to packages/storage/src/services/scheduler.ts diff --git a/apps/backend/src/upload/tus.ts b/packages/storage/src/services/tus.ts similarity index 83% rename from apps/backend/src/upload/tus.ts rename to packages/storage/src/services/tus.ts index d98c7fe..8f9a6a9 100644 --- a/apps/backend/src/upload/tus.ts +++ b/packages/storage/src/services/tus.ts @@ -1,9 +1,9 @@ import { Server, Upload } from '@repo/tus'; import { prisma } from '@repo/db'; -import { getFilenameWithoutExt } from '../utils/file'; import { nanoid } from 'nanoid'; import { slugify } from 'transliteration'; -import { StorageManager } from './storage.adapter'; +import { StorageManager } from '../core/adapter'; +import { createResource, updateResourceStatus } from '../database/operations'; const FILE_UPLOAD_CONFIG = { maxSizeBytes: 20_000_000_000, // 20GB @@ -32,20 +32,23 @@ function getFileId(uploadId: string) { return uploadId.replace(/\/[^/]+$/, ''); } +function getFilenameWithoutExt(filename: string): string { + const lastDotIndex = filename.lastIndexOf('.'); + return lastDotIndex > 0 ? filename.substring(0, lastDotIndex) : filename; +} + async function handleUploadCreate(req: any, res: any, upload: Upload, url: string) { try { const fileId = getFileId(upload.id); const storageManager = StorageManager.getInstance(); - await prisma.resource.create({ - data: { - title: getFilenameWithoutExt(upload.metadata?.filename || 'untitled'), - fileId, // 移除最后的文件名 - url: upload.id, - meta: upload.metadata, - status: ResourceStatus.UPLOADING, - storageType: storageManager.getStorageType(), // 记录存储类型 - }, + await createResource({ + fileId, + filename: upload.metadata?.filename || 'untitled', + size: upload.size || 0, + mimeType: upload.metadata?.filetype, + storageType: storageManager.getStorageType(), + status: ResourceStatus.UPLOADING, }); console.log(`Resource created for ${upload.id} using ${storageManager.getStorageType()} storage`); @@ -56,14 +59,12 @@ async function handleUploadCreate(req: any, res: any, upload: Upload, url: strin async function handleUploadFinish(req: any, res: any, upload: Upload) { try { - const resource = await prisma.resource.update({ - where: { fileId: getFileId(upload.id) }, - data: { status: ResourceStatus.UPLOADED }, - }); + const fileId = getFileId(upload.id); + await updateResourceStatus(fileId, ResourceStatus.UPLOADED); // TODO: 这里可以添加队列处理逻辑 // fileQueue.add(QueueJobType.FILE_PROCESS, { resource }, { jobId: resource.id }); - console.log(`Upload finished ${resource.url} using ${StorageManager.getInstance().getStorageType()} storage`); + console.log(`Upload finished ${upload.id} using ${StorageManager.getInstance().getStorageType()} storage`); } catch (error) { console.error('Failed to update resource after upload', error); } diff --git a/apps/backend/src/upload/storage.utils.ts b/packages/storage/src/services/utils.ts similarity index 71% rename from apps/backend/src/upload/storage.utils.ts rename to packages/storage/src/services/utils.ts index 1c37c8d..68fb5e4 100644 --- a/apps/backend/src/upload/storage.utils.ts +++ b/packages/storage/src/services/utils.ts @@ -1,4 +1,5 @@ -import { StorageManager, StorageType } from './storage.adapter'; +import { StorageManager } from '../core/adapter'; +import { StorageType } from '../types'; import path from 'path'; /** @@ -20,37 +21,46 @@ export class StorageUtils { } /** - * 生成文件访问URL + * 生成文件访问URL(统一使用下载接口) * @param fileId 文件ID - * @param isPublic 是否为公开访问链接 + * @param baseUrl 基础URL(可选,用于生成完整URL) * @returns 文件访问URL */ - public generateFileUrl(fileId: string, isPublic: boolean = false): string { + public generateFileUrl(fileId: string, baseUrl?: string): string { + const base = baseUrl || 'http://localhost:3000'; + return `${base}/download/${fileId}`; + } + + /** + * 生成文件下载URL(与 generateFileUrl 相同,保持兼容性) + * @param fileId 文件ID + * @param baseUrl 基础URL(可选,用于生成完整URL) + * @returns 文件下载URL + */ + public generateDownloadUrl(fileId: string, baseUrl?: string): string { + return this.generateFileUrl(fileId, baseUrl); + } + + /** + * 生成直接访问URL(仅用于S3存储) + * @param fileId 文件ID + * @returns S3直接访问URL + */ + public generateDirectUrl(fileId: string): string { const storageType = this.storageManager.getStorageType(); const config = this.storageManager.getStorageConfig(); - switch (storageType) { - case StorageType.LOCAL: - // 本地存储返回相对路径或服务器路径 - if (isPublic) { - // 假设有一个静态文件服务 - return `/uploads/${fileId}`; - } - return path.join(config.local?.directory || './uploads', fileId); - - case StorageType.S3: - // S3 存储返回对象存储路径 - const s3Config = config.s3!; - if (s3Config.endpoint && s3Config.endpoint !== 'https://s3.amazonaws.com') { - // 自定义 S3 兼容服务 - return `${s3Config.endpoint}/${s3Config.bucket}/${fileId}`; - } - // AWS S3 - return `https://${s3Config.bucket}.s3.${s3Config.region}.amazonaws.com/${fileId}`; - - default: - throw new Error(`Unsupported storage type: ${storageType}`); + if (storageType !== StorageType.S3) { + throw new Error('Direct URL is only available for S3 storage'); } + + const s3Config = config.s3!; + if (s3Config.endpoint && s3Config.endpoint !== 'https://s3.amazonaws.com') { + // 自定义 S3 兼容服务 + return `${s3Config.endpoint}/${s3Config.bucket}/${fileId}`; + } + // AWS S3 + return `https://${s3Config.bucket}.s3.${s3Config.region}.amazonaws.com/${fileId}`; } /** @@ -199,4 +209,29 @@ export class StorageUtils { return stats; } + + /** + * 清理过期文件 + */ + public async cleanupExpiredFiles(): Promise<{ deletedCount: number }> { + const storageType = this.storageManager.getStorageType(); + const config = this.storageManager.getStorageConfig(); + let deletedCount = 0; + + // 获取过期时间配置 + const expirationMs = + storageType === StorageType.LOCAL + ? config.local?.expirationPeriodInMilliseconds + : config.s3?.expirationPeriodInMilliseconds; + + if (!expirationMs || expirationMs <= 0) { + // 没有配置过期时间,不执行清理 + return { deletedCount: 0 }; + } + + // TODO: 实现具体的清理逻辑 + // 这里需要根据存储类型和数据库记录来清理过期文件 + + return { deletedCount }; + } } diff --git a/packages/storage/src/types/index.ts b/packages/storage/src/types/index.ts new file mode 100644 index 0000000..2c6d6cc --- /dev/null +++ b/packages/storage/src/types/index.ts @@ -0,0 +1,51 @@ +export interface UploadCompleteEvent { + identifier: string; + filename: string; + size: number; + hash: string; + integrityVerified: boolean; +} + +export type UploadEvent = { + uploadStart: { + identifier: string; + filename: string; + totalSize: number; + resuming?: boolean; + }; + uploadComplete: UploadCompleteEvent; + uploadError: { identifier: string; error: string; filename: string }; +}; + +export interface UploadLock { + clientId: string; + timestamp: number; +} + +// 存储类型枚举 +export enum StorageType { + LOCAL = 'local', + S3 = 's3', +} + +// 存储配置接口 +export interface StorageConfig { + type: StorageType; + // 本地存储配置 + local?: { + directory: string; + expirationPeriodInMilliseconds?: number; + }; + // S3 存储配置 + s3?: { + bucket: string; + region: string; + accessKeyId: string; + secretAccessKey: string; + endpoint?: string; // 用于兼容其他 S3 兼容服务 + forcePathStyle?: boolean; + partSize?: number; + maxConcurrentPartUploads?: number; + expirationPeriodInMilliseconds?: number; + }; +} diff --git a/packages/storage/tsconfig.json b/packages/storage/tsconfig.json new file mode 100644 index 0000000..3a8900e --- /dev/null +++ b/packages/storage/tsconfig.json @@ -0,0 +1,29 @@ +{ + "extends": "../../tsconfig.json", + "compilerOptions": { + "outDir": "./dist", + "rootDir": "./src", + "declaration": true, + "declarationMap": true, + "sourceMap": true, + "target": "ES2020", + "module": "ESNext", + "moduleResolution": "bundler", + "allowSyntheticDefaultImports": true, + "esModuleInterop": true, + "strict": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "isolatedModules": true, + "noEmitOnError": false + }, + "include": [ + "src/**/*" + ], + "exclude": [ + "dist", + "node_modules", + "**/*.test.ts", + "**/*.spec.ts" + ] +} \ No newline at end of file diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index b55278a..7fc39a0 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -47,6 +47,9 @@ importers: '@repo/oidc-provider': specifier: workspace:* version: link:../../packages/oidc-provider + '@repo/storage': + specifier: workspace:* + version: link:../../packages/storage '@trpc/server': specifier: 11.1.2 version: 11.1.2(typescript@5.8.3) @@ -65,9 +68,6 @@ importers: nanoid: specifier: ^5.1.5 version: 5.1.5 - nanoid-cjs: - specifier: ^0.0.7 - version: 0.0.7 node-cron: specifier: ^4.0.7 version: 4.0.7 @@ -98,7 +98,7 @@ importers: version: 7.1.1 vitest: specifier: ^3.1.4 - version: 3.1.4(@types/node@22.15.21)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.19.4)(yaml@2.8.0) + version: 3.1.4(@types/debug@4.1.12)(@types/node@22.15.21)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.19.4)(yaml@2.8.0) apps/web: dependencies: @@ -165,6 +165,9 @@ importers: superjson: specifier: ^2.2.2 version: 2.2.2 + tus-js-client: + specifier: ^4.3.1 + version: 4.3.1 valibot: specifier: ^1.1.0 version: 1.1.0(typescript@5.8.3) @@ -436,6 +439,43 @@ importers: specifier: ^5.8.3 version: 5.8.3 + packages/storage: + dependencies: + '@hono/zod-validator': + specifier: ^0.5.0 + version: 0.5.0(hono@4.7.10)(zod@3.25.23) + '@repo/db': + specifier: workspace:* + version: link:../db + '@repo/tus': + specifier: workspace:* + version: link:../tus + hono: + specifier: ^4.7.10 + version: 4.7.10 + ioredis: + specifier: 5.4.1 + version: 5.4.1 + jose: + specifier: ^6.0.11 + version: 6.0.11 + nanoid: + specifier: ^5.1.5 + version: 5.1.5 + transliteration: + specifier: ^2.3.5 + version: 2.3.5 + zod: + specifier: ^3.25.23 + version: 3.25.23 + devDependencies: + '@types/node': + specifier: ^22.15.21 + version: 22.15.21 + typescript: + specifier: ^5.0.0 + version: 5.8.3 + packages/tus: dependencies: '@aws-sdk/client-s3': @@ -3291,10 +3331,6 @@ packages: resolution: {integrity: sha512-ZmdL2rui+eB2YwhsWzjInR8LldtZHGDoQ1ugH85ppHKwpUHL7j7rN0Ti9NCnGiQbhaZ11FpR+7ao1dNsmduNUg==} engines: {node: '>=12'} - dotenv@16.5.0: - resolution: {integrity: sha512-m/C+AwOAr9/W1UOIZUo232ejMNnJAJtYQjUbHoNTBNTJSvqzzDh7vnrei3o3r3m9blf6ZoDkvcw0VmozNRFJxg==} - engines: {node: '>=12'} - dunder-proto@1.0.1: resolution: {integrity: sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==} engines: {node: '>= 0.4'} @@ -3522,6 +3558,10 @@ packages: fast-safe-stringify@2.1.1: resolution: {integrity: sha512-W+KJc2dmILlPplD/H4K9l9LcAHAfPtP6BY84uVLXQ6Evcz9Lcg33Y2z1IVblT6xdY54PXYVHEv+0Wpq8Io6zkA==} + fast-xml-parser@4.4.1: + resolution: {integrity: sha512-xkjOecfnKGkSsOwtZ5Pz7Us/T6mrbPQrq0nh+aCO5V9nk5NLWmasAHumTKjiPJPWANe+kAZ84Jc8ooJkzZ88Sw==} + hasBin: true + fast-xml-parser@4.5.3: resolution: {integrity: sha512-RKihhV+SHsIUGXObeVy9AXiBbFwkVk7Syp8XgwN5U3JV416+Gwp/GO9i0JYKmikykgz/UHRrrV4ROuZEo/T0ig==} hasBin: true @@ -4434,9 +4474,6 @@ packages: mz@2.7.0: resolution: {integrity: sha512-z81GNO7nnYMEhrGh9LeymoE4+Yr0Wn5McHIZMK5cfQCl+NDX08sCZgUc9/6MHni9IWuFLm1Z3HTCXu2z9fN62Q==} - nanoid-cjs@0.0.7: - resolution: {integrity: sha512-z72crZ0JcTb5s40Pm9Vk99qfEw9Oe1qyVjK/kpelCKyZDH8YTX4HejSfp54PMJT8F5rmsiBpG6wfVAGAhLEFhA==} - nanoid@3.3.11: resolution: {integrity: sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w==} engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} @@ -8859,8 +8896,6 @@ snapshots: dotenv@16.4.5: {} - dotenv@16.5.0: {} - dunder-proto@1.0.1: dependencies: call-bind-apply-helpers: 1.0.2 @@ -9273,6 +9308,10 @@ snapshots: fast-safe-stringify@2.1.1: {} + fast-xml-parser@4.4.1: + dependencies: + strnum: 1.1.2 + fast-xml-parser@4.5.3: dependencies: strnum: 1.1.2 @@ -10207,10 +10246,6 @@ snapshots: object-assign: 4.1.1 thenify-all: 1.6.0 - nanoid-cjs@0.0.7: - dependencies: - nanoid: 5.1.5 - nanoid@3.3.11: {} nanoid@5.1.5: {} @@ -11561,7 +11596,7 @@ snapshots: tsx: 4.19.4 yaml: 2.8.0 - vitest@3.1.4(@types/node@22.15.21)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.19.4)(yaml@2.8.0): + vitest@3.1.4(@types/debug@4.1.12)(@types/node@22.15.21)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.19.4)(yaml@2.8.0): dependencies: '@vitest/expect': 3.1.4 '@vitest/mocker': 3.1.4(vite@6.3.5(@types/node@22.15.21)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.19.4)(yaml@2.8.0)) @@ -11585,6 +11620,7 @@ snapshots: vite-node: 3.1.4(@types/node@22.15.21)(jiti@2.4.2)(lightningcss@1.30.1)(tsx@4.19.4)(yaml@2.8.0) why-is-node-running: 2.3.0 optionalDependencies: + '@types/debug': 4.1.12 '@types/node': 22.15.21 transitivePeerDependencies: - jiti diff --git a/uploads/2025/05/28/1mVGC8r6jy/gerber_cake_pcb_cake_2025-05-12.zip b/uploads/2025/05/28/1mVGC8r6jy/gerber_cake_pcb_cake_2025-05-12.zip new file mode 100644 index 0000000..e69de29 diff --git a/uploads/2025/05/28/1mVGC8r6jy/gerber_cake_pcb_cake_2025-05-12.zip.json b/uploads/2025/05/28/1mVGC8r6jy/gerber_cake_pcb_cake_2025-05-12.zip.json new file mode 100644 index 0000000..7eb977d --- /dev/null +++ b/uploads/2025/05/28/1mVGC8r6jy/gerber_cake_pcb_cake_2025-05-12.zip.json @@ -0,0 +1 @@ +{"id":"2025/05/28/1mVGC8r6jy/gerber_cake_pcb_cake_2025-05-12.zip","metadata":{"filename":"Gerber_CAKE_PCB_CAKE_2025-05-12.zip","filetype":"application/zip"},"size":182599,"offset":0,"creation_date":"2025-05-28T02:57:41.355Z"} \ No newline at end of file diff --git a/uploads/2025/05/28/1vwaAK1QGH/ren-yuan-shu-ju_2025-04-10_13-14.xlsx b/uploads/2025/05/28/1vwaAK1QGH/ren-yuan-shu-ju_2025-04-10_13-14.xlsx new file mode 100644 index 0000000..e6b8051 Binary files /dev/null and b/uploads/2025/05/28/1vwaAK1QGH/ren-yuan-shu-ju_2025-04-10_13-14.xlsx differ diff --git a/uploads/2025/05/28/1vwaAK1QGH/ren-yuan-shu-ju_2025-04-10_13-14.xlsx.json b/uploads/2025/05/28/1vwaAK1QGH/ren-yuan-shu-ju_2025-04-10_13-14.xlsx.json new file mode 100644 index 0000000..ab2c029 --- /dev/null +++ b/uploads/2025/05/28/1vwaAK1QGH/ren-yuan-shu-ju_2025-04-10_13-14.xlsx.json @@ -0,0 +1 @@ +{"id":"2025/05/28/1vwaAK1QGH/ren-yuan-shu-ju_2025-04-10_13-14.xlsx","metadata":{"filename":"人员数据_2025-04-10_13-14.xlsx","filetype":"application/wps-office.xlsx"},"size":14371,"offset":0,"creation_date":"2025-05-28T03:31:11.834Z"} \ No newline at end of file diff --git a/uploads/2025/05/28/8GPTrrQBkM/icon.png b/uploads/2025/05/28/8GPTrrQBkM/icon.png new file mode 100644 index 0000000..69a1a55 Binary files /dev/null and b/uploads/2025/05/28/8GPTrrQBkM/icon.png differ diff --git a/uploads/2025/05/28/8GPTrrQBkM/icon.png.json b/uploads/2025/05/28/8GPTrrQBkM/icon.png.json new file mode 100644 index 0000000..43f1a5c --- /dev/null +++ b/uploads/2025/05/28/8GPTrrQBkM/icon.png.json @@ -0,0 +1 @@ +{"id":"2025/05/28/8GPTrrQBkM/icon.png","metadata":{"filename":"icon.png","filetype":"image/png"},"size":4820,"offset":0,"creation_date":"2025-05-28T03:30:46.057Z"} \ No newline at end of file diff --git a/uploads/2025/05/28/D_o4NakhA_/localmaven-1-.zip b/uploads/2025/05/28/D_o4NakhA_/localmaven-1-.zip new file mode 100644 index 0000000..8b54f7a Binary files /dev/null and b/uploads/2025/05/28/D_o4NakhA_/localmaven-1-.zip differ diff --git a/uploads/2025/05/28/D_o4NakhA_/localmaven-1-.zip.json b/uploads/2025/05/28/D_o4NakhA_/localmaven-1-.zip.json new file mode 100644 index 0000000..ef7f535 --- /dev/null +++ b/uploads/2025/05/28/D_o4NakhA_/localmaven-1-.zip.json @@ -0,0 +1 @@ +{"id":"2025/05/28/D_o4NakhA_/localmaven-1-.zip","metadata":{"filename":"localMaven (1).zip","filetype":"application/zip"},"size":9765568,"offset":0,"creation_date":"2025-05-28T09:09:41.250Z"} \ No newline at end of file diff --git a/uploads/2025/05/28/FbI9ERQnCB/aristocrats-build-0-0-5.rar b/uploads/2025/05/28/FbI9ERQnCB/aristocrats-build-0-0-5.rar new file mode 100644 index 0000000..83a2a5b Binary files /dev/null and b/uploads/2025/05/28/FbI9ERQnCB/aristocrats-build-0-0-5.rar differ diff --git a/uploads/2025/05/28/FbI9ERQnCB/aristocrats-build-0-0-5.rar.json b/uploads/2025/05/28/FbI9ERQnCB/aristocrats-build-0-0-5.rar.json new file mode 100644 index 0000000..14eef04 --- /dev/null +++ b/uploads/2025/05/28/FbI9ERQnCB/aristocrats-build-0-0-5.rar.json @@ -0,0 +1 @@ +{"id":"2025/05/28/FbI9ERQnCB/aristocrats-build-0-0-5.rar","metadata":{"filename":"Aristocrats Build 0-0-5.rar","filetype":"application/vnd.rar"},"size":32640420,"offset":0,"creation_date":"2025-05-28T03:30:30.904Z"} \ No newline at end of file diff --git a/uploads/2025/05/28/GCOk6TcqUp/she-bei-wang-luo-sdkshi-yong-shou-ce.chm b/uploads/2025/05/28/GCOk6TcqUp/she-bei-wang-luo-sdkshi-yong-shou-ce.chm new file mode 100644 index 0000000..f25c0eb Binary files /dev/null and b/uploads/2025/05/28/GCOk6TcqUp/she-bei-wang-luo-sdkshi-yong-shou-ce.chm differ diff --git a/uploads/2025/05/28/GCOk6TcqUp/she-bei-wang-luo-sdkshi-yong-shou-ce.chm.json b/uploads/2025/05/28/GCOk6TcqUp/she-bei-wang-luo-sdkshi-yong-shou-ce.chm.json new file mode 100644 index 0000000..97b9895 --- /dev/null +++ b/uploads/2025/05/28/GCOk6TcqUp/she-bei-wang-luo-sdkshi-yong-shou-ce.chm.json @@ -0,0 +1 @@ +{"id":"2025/05/28/GCOk6TcqUp/she-bei-wang-luo-sdkshi-yong-shou-ce.chm","metadata":{"filename":"设备网络SDK使用手册.chm","filetype":"application/vnd.ms-htmlhelp"},"size":12922557,"offset":0,"creation_date":"2025-05-28T03:22:41.186Z"} \ No newline at end of file diff --git a/uploads/2025/05/28/QPQmmZCRZw/image.jpg b/uploads/2025/05/28/QPQmmZCRZw/image.jpg new file mode 100644 index 0000000..e69de29 diff --git a/uploads/2025/05/28/QPQmmZCRZw/image.jpg.json b/uploads/2025/05/28/QPQmmZCRZw/image.jpg.json new file mode 100644 index 0000000..422de9a --- /dev/null +++ b/uploads/2025/05/28/QPQmmZCRZw/image.jpg.json @@ -0,0 +1 @@ +{"id":"2025/05/28/QPQmmZCRZw/image.jpg","metadata":{"filename":"image.jpg","filetype":"image/jpeg"},"size":226196,"offset":0,"creation_date":"2025-05-28T03:16:52.123Z"} \ No newline at end of file diff --git a/uploads/2025/05/28/RHwt8AkkZp/chao-ren-qiang.jpeg b/uploads/2025/05/28/RHwt8AkkZp/chao-ren-qiang.jpeg new file mode 100644 index 0000000..9876b4e Binary files /dev/null and b/uploads/2025/05/28/RHwt8AkkZp/chao-ren-qiang.jpeg differ diff --git a/uploads/2025/05/28/RHwt8AkkZp/chao-ren-qiang.jpeg.json b/uploads/2025/05/28/RHwt8AkkZp/chao-ren-qiang.jpeg.json new file mode 100644 index 0000000..00af59c --- /dev/null +++ b/uploads/2025/05/28/RHwt8AkkZp/chao-ren-qiang.jpeg.json @@ -0,0 +1 @@ +{"id":"2025/05/28/RHwt8AkkZp/chao-ren-qiang.jpeg","metadata":{"filename":"超人强.jpeg","filetype":"image/jpeg"},"size":8133,"offset":0,"creation_date":"2025-05-28T08:28:19.786Z"} \ No newline at end of file diff --git a/uploads/2025/05/28/SSw-o3MpWO/x4_asm.3mf b/uploads/2025/05/28/SSw-o3MpWO/x4_asm.3mf new file mode 100644 index 0000000..b086ecc Binary files /dev/null and b/uploads/2025/05/28/SSw-o3MpWO/x4_asm.3mf differ diff --git a/uploads/2025/05/28/SSw-o3MpWO/x4_asm.3mf.json b/uploads/2025/05/28/SSw-o3MpWO/x4_asm.3mf.json new file mode 100644 index 0000000..cc42716 --- /dev/null +++ b/uploads/2025/05/28/SSw-o3MpWO/x4_asm.3mf.json @@ -0,0 +1 @@ +{"id":"2025/05/28/SSw-o3MpWO/x4_asm.3mf","metadata":{"filename":"x4_asm.3mf","filetype":null},"size":1375612,"offset":0,"creation_date":"2025-05-28T03:30:59.326Z"} \ No newline at end of file diff --git a/uploads/2025/05/28/YHcSbzTuMP/image-4-.jpg b/uploads/2025/05/28/YHcSbzTuMP/image-4-.jpg new file mode 100644 index 0000000..dc1d1c7 Binary files /dev/null and b/uploads/2025/05/28/YHcSbzTuMP/image-4-.jpg differ diff --git a/uploads/2025/05/28/YHcSbzTuMP/image-4-.jpg.json b/uploads/2025/05/28/YHcSbzTuMP/image-4-.jpg.json new file mode 100644 index 0000000..c079be6 --- /dev/null +++ b/uploads/2025/05/28/YHcSbzTuMP/image-4-.jpg.json @@ -0,0 +1 @@ +{"id":"2025/05/28/YHcSbzTuMP/image-4-.jpg","metadata":{"filename":"image (4).jpg","filetype":"image/jpeg"},"size":299358,"offset":0,"creation_date":"2025-05-28T03:22:25.465Z"} \ No newline at end of file diff --git a/uploads/2025/05/28/_CfOJ3MDpx/bao-xiao-zhi-fu-ji-lu.jpg b/uploads/2025/05/28/_CfOJ3MDpx/bao-xiao-zhi-fu-ji-lu.jpg new file mode 100644 index 0000000..45b154b Binary files /dev/null and b/uploads/2025/05/28/_CfOJ3MDpx/bao-xiao-zhi-fu-ji-lu.jpg differ diff --git a/uploads/2025/05/28/_CfOJ3MDpx/bao-xiao-zhi-fu-ji-lu.jpg.json b/uploads/2025/05/28/_CfOJ3MDpx/bao-xiao-zhi-fu-ji-lu.jpg.json new file mode 100644 index 0000000..4f4745d --- /dev/null +++ b/uploads/2025/05/28/_CfOJ3MDpx/bao-xiao-zhi-fu-ji-lu.jpg.json @@ -0,0 +1 @@ +{"id":"2025/05/28/_CfOJ3MDpx/bao-xiao-zhi-fu-ji-lu.jpg","metadata":{"filename":"报销支付记录.jpg","filetype":"image/jpeg"},"size":303648,"offset":0,"creation_date":"2025-05-28T09:07:09.099Z"} \ No newline at end of file diff --git a/uploads/2025/05/28/bPEBQAcXdC/image-3-.jpg b/uploads/2025/05/28/bPEBQAcXdC/image-3-.jpg new file mode 100644 index 0000000..c39e687 Binary files /dev/null and b/uploads/2025/05/28/bPEBQAcXdC/image-3-.jpg differ diff --git a/uploads/2025/05/28/bPEBQAcXdC/image-3-.jpg.json b/uploads/2025/05/28/bPEBQAcXdC/image-3-.jpg.json new file mode 100644 index 0000000..78cd177 --- /dev/null +++ b/uploads/2025/05/28/bPEBQAcXdC/image-3-.jpg.json @@ -0,0 +1 @@ +{"id":"2025/05/28/bPEBQAcXdC/image-3-.jpg","metadata":{"filename":"image (3).jpg","filetype":"image/jpeg"},"size":275492,"offset":0,"creation_date":"2025-05-28T03:39:37.972Z"} \ No newline at end of file diff --git a/uploads/2025/05/28/e51n6saF8T/aristocrats-build-0-0-5.rar b/uploads/2025/05/28/e51n6saF8T/aristocrats-build-0-0-5.rar new file mode 100644 index 0000000..66cef27 Binary files /dev/null and b/uploads/2025/05/28/e51n6saF8T/aristocrats-build-0-0-5.rar differ diff --git a/uploads/2025/05/28/e51n6saF8T/aristocrats-build-0-0-5.rar.json b/uploads/2025/05/28/e51n6saF8T/aristocrats-build-0-0-5.rar.json new file mode 100644 index 0000000..f5b7230 --- /dev/null +++ b/uploads/2025/05/28/e51n6saF8T/aristocrats-build-0-0-5.rar.json @@ -0,0 +1 @@ +{"id":"2025/05/28/e51n6saF8T/aristocrats-build-0-0-5.rar","metadata":{"filename":"Aristocrats Build 0-0-5.rar","filetype":"application/vnd.rar"},"size":32640420,"offset":0,"creation_date":"2025-05-28T03:23:41.183Z"} \ No newline at end of file diff --git a/uploads/2025/05/28/smT7WbJukS/shi-ke-zhang-1-.3mf b/uploads/2025/05/28/smT7WbJukS/shi-ke-zhang-1-.3mf new file mode 100644 index 0000000..3029db2 Binary files /dev/null and b/uploads/2025/05/28/smT7WbJukS/shi-ke-zhang-1-.3mf differ diff --git a/uploads/2025/05/28/smT7WbJukS/shi-ke-zhang-1-.3mf.json b/uploads/2025/05/28/smT7WbJukS/shi-ke-zhang-1-.3mf.json new file mode 100644 index 0000000..1d63a84 --- /dev/null +++ b/uploads/2025/05/28/smT7WbJukS/shi-ke-zhang-1-.3mf.json @@ -0,0 +1 @@ +{"id":"2025/05/28/smT7WbJukS/shi-ke-zhang-1-.3mf","metadata":{"filename":"蚀刻章(1).3mf","filetype":null},"size":1722342,"offset":0,"creation_date":"2025-05-28T03:20:56.034Z"} \ No newline at end of file