バグ #267
未完了Phase C 詳細実装仕様書: Advanced RAG & Vector Search 実装
0%
説明
Phase C 詳細実装仕様書¶
Advanced RAG & Vector Search Implementation¶
🎯 実装範囲の確認¶
既存実装状況 (2025-06-05調査結果):
- ✅ Vector Database: PostgreSQL + pgvector (task2-vector-db)
- ✅ Redis Cache: task2-redis (キャッシュサービス)
- ✅ Search Engine: Meilisearch (task2-search)
- ✅ Basic RAG Pipeline: 基本的なembeddings/documents機能
- ✅ File Processing: multer, pdf-parse, mammoth実装済み
- ✅ OpenAI Integration: openai, tiktoken実装済み
Phase C追加実装対象:
- Advanced RAG Pipeline - 高度な検索・回答生成
- Multi-Modal Processing - 複数ファイル形式対応
- Performance Optimization - 並列処理・キャッシュ最適化
- Chat Interface - RAG統合チャットシステム
- Analytics & Monitoring - 使用統計・パフォーマンス監視
📚 関数仕様・API設計¶
1. VectorDatabaseService¶
1.1 createDocument(documentData)
目的: 新しいドキュメントをベクトルDB投入
入力:
{
title: string,
content: string,
metadata: {
fileType: string,
originalName: string,
size: number,
tags: string[]
}
}
出力:
{
id: number,
title: string,
chunks: number,
embeddings: number,
status: 'processing' | 'completed' | 'failed'
}
エラー制御:
-
ValidationError
: 入力データ不正 -
DatabaseError
: DB接続・保存エラー -
EmbeddingError
: 埋め込み生成失敗
1.2 semanticSearch(query, options)
目的: セマンティック検索実行
入力:
query: string, // 検索クエリ
options: {
limit: number = 10,
threshold: number = 0.7,
filters: {
dateRange?: [Date, Date],
tags?: string[],
fileTypes?: string[]
}
}
出力:
{
results: [{
documentId: number,
chunkId: number,
content: string,
similarity: number,
metadata: object
}],
totalResults: number,
searchTime: number
}
1.3 updateEmbeddings(documentId)
目的: ドキュメント埋め込み再生成
入力: documentId: number
出力: { success: boolean, chunksUpdated: number }
2. RAGQueryService¶
2.1 generateAnswer(query, context)
目的: コンテキスト基づく回答生成
入力:
{
query: string,
context: [{
content: string,
source: string,
relevance: number
}],
options: {
model: 'gpt-4' | 'gpt-3.5-turbo',
maxTokens: number = 1000,
temperature: number = 0.3
}
}
出力:
{
answer: string,
sources: string[],
confidence: number,
tokens: {
prompt: number,
completion: number,
total: number
}
}
2.2 buildContext(searchResults, query)
目的: 検索結果から最適なコンテキスト構築
入力:
-
searchResults
: semanticSearch結果 -
query
: string
出力: 最適化されたコンテキスト配列
3. DocumentProcessingService¶
3.1 processFile(file, processingOptions)
目的: アップロードファイルの処理・分析
入力:
file: Express.Multer.File,
processingOptions: {
chunkSize: number = 1000,
overlap: number = 200,
extractMetadata: boolean = true,
generateSummary: boolean = true
}
出力:
{
document: {
id: number,
title: string,
summary: string
},
chunks: [{
id: number,
content: string,
position: number
}],
metadata: {
pages: number,
words: number,
language: string,
extractedImages: number
}
}
3.2 extractContent(buffer, fileType)
目的: ファイル内容抽出(PDF, DOCX, TXT等)
入力:
-
buffer
: Buffer -
fileType
: string
出力:{ content: string, metadata: object }
4. CacheService¶
4.1 getEmbedding(text)
目的: 埋め込みキャッシュ取得
入力: text: string
出力: number[] | null
4.2 setEmbedding(text, embedding, ttl)
目的: 埋め込みキャッシュ設定
入力:
text: string
embedding: number[]
-
ttl: number = 3600
出力:boolean
5. AnalyticsService¶
5.1 trackSearch(query, results, userId)
目的: 検索統計記録
入力: 検索クエリ、結果、ユーザーID
出力: { tracked: boolean }
5.2 getSearchAnalytics(timeRange)
目的: 検索統計取得
入力: timeRange: { start: Date, end: Date }
出力: 統計データオブジェクト
🏗️ API エンドポイント設計¶
1. Document Management API¶
POST /api/documents/upload
¶
認証: Required
Content-Type: multipart/form-data
Request:
{
file: File, // アップロードファイル
title?: string,
tags?: string[],
processingOptions?: {
chunkSize: number,
generateSummary: boolean
}
}
Response:
{
success: true,
document: {
id: number,
title: string,
status: string,
uploadedAt: string
},
processing: {
jobId: string,
estimatedTime: number
}
}
GET /api/documents
¶
認証: Required
Query Parameters:
-
page
: number = 1 -
limit
: number = 20 -
search
: string -
tags
: string[] -
sortBy
: 'created' | 'title' | 'size'
DELETE /api/documents/:id
¶
認証: Required
Response: { success: boolean, deleted: boolean }
2. Search API¶
POST /api/search/semantic
¶
認証: Required
Request:
{
query: string,
options?: {
limit: number,
threshold: number,
filters: object
}
}
Response:
{
results: SearchResult[],
metadata: {
totalResults: number,
searchTime: number,
queryEmbeddingTime: number
}
}
3. Chat API¶
POST /api/chat/message
¶
認証: Required
Request:
{
message: string,
sessionId?: string,
context?: {
useRAG: boolean = true,
maxSources: number = 5
}
}
Response:
{
response: string,
sessionId: string,
sources: [{
title: string,
content: string,
relevance: number
}],
metadata: {
model: string,
tokensUsed: number,
processingTime: number
}
}
🔧 依存関係・インストール順序¶
1. NPM Packages 追加インストール¶
# Phase C 新規パッケージ(優先順)
npm install --save \
# 1. Core Dependencies
openai@^4.20.1 \
tiktoken@^1.0.10 \
# 2. File Processing
multer@^1.4.5-lts.1 \
pdf-parse@^1.1.1 \
mammoth@^1.6.0 \
node-html-markdown@^1.3.0 \
# 3. Vector & Search
pgvector@^0.1.8 \
meilisearch@^0.37.0 \
# 4. Cache & Session
redis@^4.6.8 \
express-session@^1.17.3 \
# 5. Utilities
uuid@^9.0.1 \
lodash@^4.17.21 \
moment@^2.29.4
2. Development Dependencies¶
npm install --save-dev \
jest@^29.7.0 \
supertest@^6.3.3 \
@types/jest@^29.5.8 \
@types/multer@^1.4.11
3. Docker Services 起動順序¶
# 1. Infrastructure Services
docker-compose up -d task2-vector-db
docker-compose up -d task2-redis
docker-compose up -d task2-search
# 2. Wait for services to be ready
sleep 30
# 3. Application Services
docker-compose up -d task2-api
docker-compose up -d task2-ui
🏢 機能単位開発グループ¶
Group A: Core Infrastructure (1.5時間)¶
責任範囲: データベース・キャッシュ・基盤機能
実装対象:
- VectorDatabaseService 完全実装
- CacheService (Redis統合)
- Database Migration/Schema
- Health Check強化
担当ファイル:
src/services/VectorDatabaseService.js
src/services/CacheService.js
src/db/migrations/
src/health/
Group B: Document Processing (2時間)¶
責任範囲: ファイル処理・埋め込み生成
実装対象:
- DocumentProcessingService
- File upload handling
- Content extraction pipeline
- Chunking & embedding generation
担当ファイル:
src/services/DocumentProcessingService.js
src/controllers/documentsController.js
src/middleware/uploadMiddleware.js
src/utils/contentExtractor.js
Group C: Search & RAG Engine (2時間)¶
責任範囲: 検索・回答生成エンジン
実装対象:
- RAGQueryService
- SearchController
- Query optimization
- Context building
担当ファイル:
src/services/RAGQueryService.js
src/controllers/searchController.js
src/utils/queryOptimizer.js
src/services/ContextBuilder.js
Group D: Chat Interface (1.5時間)¶
責任範囲: チャット機能・ユーザーインターフェース
実装対象:
- ChatController
- Session management
- Real-time communication
- Frontend integration
担当ファイル:
src/controllers/chatController.js
src/services/ChatSessionService.js
src/routes/chat.js
- Frontend chat components
Group E: Analytics & Optimization (1時間)¶
責任範囲: 統計・監視・最適化
実装対象:
- AnalyticsService
- Performance monitoring
- Usage statistics
- Optimization algorithms
担当ファイル:
src/services/AnalyticsService.js
src/middleware/analyticsMiddleware.js
src/utils/performanceOptimizer.js
🔍 エラー制御・例外処理¶
Error Hierarchy¶
class RAGError extends Error {
constructor(message, code, details) {
super(message);
this.name = 'RAGError';
this.code = code;
this.details = details;
}
}
class VectorDatabaseError extends RAGError {}
class EmbeddingError extends RAGError {}
class DocumentProcessingError extends RAGError {}
class SearchError extends RAGError {}
Global Error Handler¶
// src/middleware/errorHandler.js
function errorHandler(err, req, res, next) {
if (err instanceof RAGError) {
return res.status(400).json({
error: err.name,
message: err.message,
code: err.code,
details: err.details
});
}
// Default error handling
res.status(500).json({
error: 'InternalServerError',
message: 'Something went wrong'
});
}
📊 テスト戦略¶
Unit Tests (60%+ coverage目標)¶
// 例: VectorDatabaseService.test.js
describe('VectorDatabaseService', () => {
test('should create document with valid data', async () => {
const result = await vectorService.createDocument(validDocData);
expect(result.id).toBeDefined();
expect(result.status).toBe('processing');
});
test('should perform semantic search', async () => {
const results = await vectorService.semanticSearch('test query');
expect(results.results).toBeArray();
expect(results.searchTime).toBeLessThan(2000);
});
});
Integration Tests¶
- API endpoint testing
- Database integration
- File processing pipeline
- End-to-end RAG workflow
Performance Tests¶
- Load testing (50+ concurrent users)
- Response time verification (< 2 seconds)
- Memory usage monitoring
🚀 実装開始チェックリスト¶
Pre-implementation¶
- 現在のサービス状態確認
- データベーススキーマ検証
- 環境変数設定確認
- 既存コード理解・分析
Phase C Implementation¶
- Group A: Core Infrastructure
- Group B: Document Processing
- Group C: Search & RAG Engine
- Group D: Chat Interface
- Group E: Analytics & Optimization
Post-implementation¶
- Unit test execution
- Integration test execution
- Performance test execution
- Documentation update
- Deployment verification
この詳細仕様書により、Phase C実装が体系的・効率的に進行可能になります! 🎯
Redmine Admin さんが3日前に更新
🧪 Phase C テストケース仕様¶
1. VectorDatabaseService テストケース¶
1.1 createDocument()
テスト¶
describe('VectorDatabaseService.createDocument', () => {
// 正常系テスト
test('should create document with valid data', async () => {
const validDoc = {
title: 'Test Document',
content: 'This is test content for embedding generation.',
metadata: { fileType: 'txt', size: 1024 }
};
const result = await vectorService.createDocument(validDoc);
expect(result.id).toBeGreaterThan(0);
expect(result.status).toBe('processing');
expect(result.chunks).toBeGreaterThan(0);
});
// 境界値テスト
test('should handle large documents (>10MB)', async () => {
const largeDoc = { title: 'Large Doc', content: 'x'.repeat(10485760) };
const result = await vectorService.createDocument(largeDoc);
expect(result.chunks).toBeGreaterThan(100);
});
// 異常系テスト
test('should reject empty content', async () => {
const invalidDoc = { title: 'Empty', content: '' };
await expect(vectorService.createDocument(invalidDoc))
.rejects.toThrow('ValidationError');
});
test('should reject missing title', async () => {
const invalidDoc = { content: 'Content without title' };
await expect(vectorService.createDocument(invalidDoc))
.rejects.toThrow('ValidationError');
});
});
1.2 semanticSearch()
テスト¶
describe('VectorDatabaseService.semanticSearch', () => {
beforeEach(async () => {
// テストデータ投入
await seedTestDocuments();
});
// 正常系テスト
test('should return relevant results for valid query', async () => {
const results = await vectorService.semanticSearch('machine learning');
expect(results.results).toHaveLength(10);
expect(results.results[0].similarity).toBeGreaterThan(0.7);
expect(results.searchTime).toBeLessThan(2000);
});
// パフォーマンステスト
test('should complete search within 2 seconds', async () => {
const startTime = Date.now();
await vectorService.semanticSearch('complex technical query');
const duration = Date.now() - startTime;
expect(duration).toBeLessThan(2000);
});
// フィルタリングテスト
test('should apply date range filters correctly', async () => {
const options = {
filters: {
dateRange: [new Date('2025-01-01'), new Date('2025-06-01')]
}
};
const results = await vectorService.semanticSearch('test', options);
results.results.forEach(result => {
expect(new Date(result.metadata.createdAt))
.toBeWithinRange(options.filters.dateRange[0], options.filters.dateRange[1]);
});
});
// 異常系テスト
test('should handle empty query gracefully', async () => {
const results = await vectorService.semanticSearch('');
expect(results.results).toHaveLength(0);
});
test('should handle non-existent language queries', async () => {
const results = await vectorService.semanticSearch('xyz123非存在クエリ');
expect(results.results.length).toBeLessThanOrEqual(5);
});
});
2. DocumentProcessingService テストケース¶
2.1 processFile()
テスト¶
describe('DocumentProcessingService.processFile', () => {
// PDF処理テスト
test('should process PDF file correctly', async () => {
const pdfBuffer = await readTestFile('sample.pdf');
const mockFile = createMockFile(pdfBuffer, 'sample.pdf', 'application/pdf');
const result = await docService.processFile(mockFile);
expect(result.document.title).toBe('sample.pdf');
expect(result.chunks.length).toBeGreaterThan(0);
expect(result.metadata.pages).toBeGreaterThan(0);
});
// DOCX処理テスト
test('should process DOCX file correctly', async () => {
const docxBuffer = await readTestFile('sample.docx');
const mockFile = createMockFile(docxBuffer, 'sample.docx', 'application/vnd.openxmlformats-officedocument.wordprocessingml.document');
const result = await docService.processFile(mockFile);
expect(result.document.summary).toBeDefined();
expect(result.metadata.words).toBeGreaterThan(0);
});
// チャンク分割テスト
test('should chunk content with proper overlap', async () => {
const longContent = 'word '.repeat(2000);
const mockFile = createMockFile(Buffer.from(longContent), 'long.txt', 'text/plain');
const result = await docService.processFile(mockFile, {
chunkSize: 1000,
overlap: 200
});
expect(result.chunks.length).toBeGreaterThan(1);
// オーバーラップ検証
expect(result.chunks[1].content).toContain(
result.chunks[0].content.slice(-200)
);
});
// ファイルサイズ制限テスト
test('should reject files larger than limit', async () => {
const largeBuffer = Buffer.alloc(100 * 1024 * 1024); // 100MB
const mockFile = createMockFile(largeBuffer, 'large.pdf', 'application/pdf');
await expect(docService.processFile(mockFile))
.rejects.toThrow('File size exceeds limit');
});
// 不正ファイル形式テスト
test('should reject unsupported file types', async () => {
const mockFile = createMockFile(Buffer.from('content'), 'file.xyz', 'application/unknown');
await expect(docService.processFile(mockFile))
.rejects.toThrow('Unsupported file type');
});
});
3. RAGQueryService テストケース¶
3.1 generateAnswer()
テスト¶
describe('RAGQueryService.generateAnswer', () => {
// 正常回答生成テスト
test('should generate coherent answer with context', async () => {
const query = 'What is machine learning?';
const context = [
{ content: 'Machine learning is a subset of AI...', relevance: 0.9 },
{ content: 'ML algorithms learn from data...', relevance: 0.8 }
];
const result = await ragService.generateAnswer(query, context);
expect(result.answer).toContain('machine learning');
expect(result.confidence).toBeGreaterThan(0.7);
expect(result.sources.length).toBeGreaterThan(0);
});
// トークン制限テスト
test('should respect token limits', async () => {
const longQuery = 'explain '.repeat(1000);
const result = await ragService.generateAnswer(longQuery, [], { maxTokens: 100 });
expect(result.tokens.completion).toBeLessThanOrEqual(100);
});
// コンテキスト不足テスト
test('should handle insufficient context gracefully', async () => {
const result = await ragService.generateAnswer('specific technical question', []);
expect(result.answer).toContain('insufficient information');
expect(result.confidence).toBeLessThan(0.5);
});
});
4. API エンドポイント統合テスト¶
4.1 Document Upload API テスト¶
describe('POST /api/documents/upload', () => {
test('should upload and process document successfully', async () => {
const response = await request(app)
.post('/api/documents/upload')
.attach('file', 'test/fixtures/sample.pdf')
.field('title', 'Test Document')
.expect(200);
expect(response.body.success).toBe(true);
expect(response.body.document.id).toBeDefined();
expect(response.body.processing.jobId).toBeDefined();
});
test('should reject unauthorized requests', async () => {
await request(app)
.post('/api/documents/upload')
.attach('file', 'test/fixtures/sample.pdf')
.expect(401);
});
test('should validate file size limits', async () => {
const largeFile = Buffer.alloc(51 * 1024 * 1024); // 51MB
await request(app)
.post('/api/documents/upload')
.attach('file', largeFile, 'large.pdf')
.set('Authorization', 'Bearer ' + validToken)
.expect(413);
});
});
4.2 Semantic Search API テスト¶
describe('POST /api/search/semantic', () => {
beforeEach(async () => {
await seedSearchTestData();
});
test('should return search results with relevance scores', async () => {
const response = await request(app)
.post('/api/search/semantic')
.set('Authorization', 'Bearer ' + validToken)
.send({ query: 'artificial intelligence' })
.expect(200);
expect(response.body.results).toBeArray();
expect(response.body.results[0].similarity).toBeGreaterThan(0.5);
expect(response.body.metadata.searchTime).toBeLessThan(2000);
});
test('should apply pagination correctly', async () => {
const response = await request(app)
.post('/api/search/semantic')
.set('Authorization', 'Bearer ' + validToken)
.send({
query: 'test',
options: { limit: 5 }
})
.expect(200);
expect(response.body.results).toHaveLength(5);
});
});
4.3 Chat API テスト¶
describe('POST /api/chat/message', () => {
test('should generate RAG-enhanced response', async () => {
const response = await request(app)
.post('/api/chat/message')
.set('Authorization', 'Bearer ' + validToken)
.send({
message: 'Tell me about machine learning',
context: { useRAG: true, maxSources: 3 }
})
.expect(200);
expect(response.body.response).toBeDefined();
expect(response.body.sources.length).toBeLessThanOrEqual(3);
expect(response.body.sessionId).toBeDefined();
});
test('should maintain session continuity', async () => {
// 最初のメッセージ
const firstResponse = await request(app)
.post('/api/chat/message')
.set('Authorization', 'Bearer ' + validToken)
.send({ message: 'Hello' })
.expect(200);
const sessionId = firstResponse.body.sessionId;
// フォローアップメッセージ
const followupResponse = await request(app)
.post('/api/chat/message')
.set('Authorization', 'Bearer ' + validToken)
.send({
message: 'What did I just say?',
sessionId: sessionId
})
.expect(200);
expect(followupResponse.body.response).toContain('Hello');
});
});
5. パフォーマンステスト¶
5.1 負荷テスト¶
describe('Performance Tests', () => {
test('should handle 50 concurrent search requests', async () => {
const promises = Array.from({ length: 50 }, () =>
request(app)
.post('/api/search/semantic')
.set('Authorization', 'Bearer ' + validToken)
.send({ query: 'test query' })
);
const startTime = Date.now();
const responses = await Promise.all(promises);
const duration = Date.now() - startTime;
expect(responses.every(r => r.status === 200)).toBe(true);
expect(duration).toBeLessThan(10000); // 10秒以内
});
test('should maintain response time under load', async () => {
const batchSize = 10;
const batches = 5;
for (let i = 0; i < batches; i++) {
const batchPromises = Array.from({ length: batchSize }, () =>
request(app)
.post('/api/search/semantic')
.set('Authorization', 'Bearer ' + validToken)
.send({ query: `batch ${i} query` })
);
const startTime = Date.now();
await Promise.all(batchPromises);
const duration = Date.now() - startTime;
expect(duration / batchSize).toBeLessThan(200); // 平均200ms以下
}
});
});
6. セキュリティテスト¶
6.1 認証・認可テスト¶
describe('Security Tests', () => {
test('should reject requests without valid token', async () => {
await request(app)
.post('/api/documents/upload')
.expect(401);
});
test('should prevent SQL injection in search queries', async () => {
const maliciousQuery = "'; DROP TABLE documents; --";
const response = await request(app)
.post('/api/search/semantic')
.set('Authorization', 'Bearer ' + validToken)
.send({ query: maliciousQuery })
.expect(200);
// データベースが正常であることを確認
expect(response.body.results).toBeDefined();
});
test('should sanitize file upload inputs', async () => {
const response = await request(app)
.post('/api/documents/upload')
.set('Authorization', 'Bearer ' + validToken)
.attach('file', Buffer.from('<script>alert("xss")</script>'), '../../../etc/passwd.txt')
.expect(400);
expect(response.body.error).toContain('Invalid file name');
});
});
7. データ整合性テスト¶
7.1 並行処理テスト¶
describe('Data Consistency Tests', () => {
test('should handle concurrent document uploads', async () => {
const uploads = Array.from({ length: 5 }, (_, i) =>
request(app)
.post('/api/documents/upload')
.set('Authorization', 'Bearer ' + validToken)
.attach('file', Buffer.from(`Content ${i}`), `doc${i}.txt`)
);
const responses = await Promise.all(uploads);
const documentIds = responses.map(r => r.body.document.id);
// 重複IDがないことを確認
expect(new Set(documentIds).size).toBe(documentIds.length);
});
test('should maintain embedding consistency after updates', async () => {
const docId = await createTestDocument();
const originalEmbeddings = await getDocumentEmbeddings(docId);
await vectorService.updateEmbeddings(docId);
const updatedEmbeddings = await getDocumentEmbeddings(docId);
expect(updatedEmbeddings.length).toBe(originalEmbeddings.length);
});
});
8. エラーハンドリングテスト¶
8.1 サービス障害テスト¶
describe('Error Handling Tests', () => {
test('should handle database connection failures gracefully', async () => {
// データベース接続を一時的に切断
await disconnectDatabase();
const response = await request(app)
.post('/api/search/semantic')
.set('Authorization', 'Bearer ' + validToken)
.send({ query: 'test' })
.expect(503);
expect(response.body.error).toBe('ServiceUnavailable');
// 接続復旧
await reconnectDatabase();
});
test('should handle OpenAI API failures', async () => {
// OpenAI APIをモック化してエラーを発生させる
mockOpenAI.embeddings.create.mockRejectedValue(new Error('API Error'));
const response = await request(app)
.post('/api/search/semantic')
.set('Authorization', 'Bearer ' + validToken)
.send({ query: 'test' })
.expect(500);
expect(response.body.error).toBe('EmbeddingError');
});
});
📊 テスト実行計画¶
1. 開発フェーズ別テスト¶
- Group A完了後: VectorDatabaseService + CacheService テスト
- Group B完了後: DocumentProcessingService テスト
- Group C完了後: RAGQueryService + SearchController テスト
- Group D完了後: ChatController + API統合テスト
- Group E完了後: AnalyticsService + パフォーマンステスト
2. テスト実行コマンド¶
# 単体テスト実行
npm test
# カバレッジ付きテスト
npm run test:coverage
# 統合テスト実行
npm run test:integration
# パフォーマンステスト実行
npm run test:performance
# 全テスト実行
npm run test:all
3. 成功基準¶
- Unit Test Coverage: > 60%
- Integration Test: 全API正常動作
- Performance Test: 応答時間 < 2秒
- Load Test: 50並行ユーザー対応
- Security Test: 脆弱性0件
これらのテストケースにより、Phase C実装の品質と信頼性が保証されます! 🧪✅