RagFlow 0.25.6 架构图全集

系统分层 · 组件拓扑 · 数据模型 · 上传链路 · 检索链路 · 部署视图
源码根目录: /home/yzq/tmp/ragflow/auth/ragflow-0.25.6
前端 React API 层 / 核心服务 业务/算法 存储与中间件

1. 系统分层架构

自上而下五层:前端表现层 → API 接口层 → 业务服务层 → 算法/解析层 → 存储中间件层。每层只依赖下层。

flowchart TB subgraph L1["表现层 · web/src"] direction LR A1["上传组件
use-document-request"] A2["检索测试
testing-form"] A3["对话/Agent UI"] A4["service + hooks
knowledge-service.ts"] end subgraph L2["接口层 · api/apps/restful_apis"] direction LR B1["document_api
上传/解析"] B2["chunk_api
检索"] B3["dialog/conversation_api
对话"] end subgraph L3["业务服务层 · api/db/services"] direction LR C1["FileService"] C2["DocumentService"] C3["TaskService"] C4["DialogService"] C5["KnowledgebaseService"] end subgraph L4["算法/解析层 · rag · deepdoc"] direction LR D1["task_executor
切片调度"] D2["deepdoc
PDF/Office 解析"] D3["Dealer.retrieval
混合检索"] D4["Embedding / Rerank
rag/llm"] end subgraph L5["存储中间件层"] direction LR E1[("MinIO
对象存储")] E2[("MySQL
元数据")] E3[["Redis
队列+缓存"]] E4[("ES/Infinity
向量库")] end L1 --> L2 --> L3 --> L4 --> L5 classDef fe fill:#16321f,stroke:#7ee787,color:#d6deeb; classDef api fill:#102a44,stroke:#4ea1ff,color:#d6deeb; classDef svc fill:#2a1f3a,stroke:#d2a8ff,color:#d6deeb; classDef store fill:#3a2a18,stroke:#ffab70,color:#d6deeb; class A1,A2,A3,A4 fe; class B1,B2,B3 api; class C1,C2,C3,C4,C5 svc; class D1,D2,D3,D4 svc; class E1,E2,E3,E4 store;

2. 组件拓扑(全景)

两条主线(上传入库 / 检索召回)如何共享同一套后端组件。实线为同步调用,虚线为异步消息流。

flowchart LR subgraph FE["前端 React + Vite"] UP["上传组件"] SE["检索表单"] end subgraph API["API 层 Quart"] DOC["document_api
upload_document"] CHK["chunk_api
retrieval_test"] end subgraph SVC["业务 + 算法"] FS["FileService"] DS["DocumentService
+ queue_tasks"] EXE["task_executor
build_chunks+embedding"] RT["Dealer.retrieval"] end subgraph MODEL["模型"] EMB["Embedding 模型"] RRK["Rerank 模型"] end subgraph STORE["存储与中间件"] MINIO[("MinIO
bucket=kb_id")] MYSQL[("MySQL
document/file/task")] REDIS[["Redis 队列"]] VDB[("ES / Infinity
ragflow_tenant_id")] end UP -->|FormData POST| DOC SE -->|POST /datasets/search| CHK DOC --> FS FS -->|put 原始文件+缩略图| MINIO FS -->|insert 元数据| MYSQL DOC -.触发解析.-> DS DS -->|写 task 表| MYSQL DS -.推入队列.-> REDIS REDIS -.consumer 消费.-> EXE EXE -->|get 取回原文件| MINIO EXE -->|encode 向量化| EMB EXE -->|insert chunks| VDB CHK --> RT RT -->|encode_queries| EMB RT -->|混合检索 全文+KNN| VDB RT -.可选重排.-> RRK classDef fe fill:#16321f,stroke:#7ee787,color:#d6deeb; classDef api fill:#102a44,stroke:#4ea1ff,color:#d6deeb; classDef svc fill:#2a1f3a,stroke:#d2a8ff,color:#d6deeb; classDef store fill:#3a2a18,stroke:#ffab70,color:#d6deeb; class UP,SE fe; class DOC,CHK api; class FS,DS,EXE,RT,EMB,RRK svc; class MINIO,MYSQL,REDIS,VDB store;

3. 数据模型关系 (ER)

MySQL 核心表关系。file2document 是文件管理树与知识库文档之间的多对多桥接表。

erDiagram TENANT ||--o{ KNOWLEDGEBASE : owns KNOWLEDGEBASE ||--o{ DOCUMENT : contains DOCUMENT ||--o{ TASK : "split into" DOCUMENT ||--|| FILE2DOCUMENT : maps FILE ||--o{ FILE2DOCUMENT : maps FILE ||--o{ FILE : "parent_id (tree)" KNOWLEDGEBASE { string id PK string tenant_id FK string embd_id "embedding 模型" string parser_id string pipeline_id int doc_num "原子递增" } DOCUMENT { string id PK string kb_id FK "= MinIO bucket" string parser_id "naive/table/paper" string location "= MinIO object key" string thumbnail string content_hash "xxhash128 去重" string run "0未跑/1运行/2取消" int chunk_num int token_num float progress } FILE { string id PK string parent_id "文件夹树" string source_type "local/knowledgebase" string location } FILE2DOCUMENT { string id PK string file_id FK string document_id FK } TASK { string id PK string doc_id FK int from_page int to_page string digest "复用判定" }
作用关键字段
knowledgebase知识库(= MinIO bucket)embd_id / parser_id / doc_num
document文档元数据kb_id / location / content_hash / run
file文件管理树节点parent_id / source_type
file2document文件↔文档桥接file_id / document_id
task解析子任务(按页/行切分)doc_id / from_page / to_page

4. 文件上传与入库链路

上传只做「落原始文件 + 写元数据」,不触发解析。时序展示同步落库全过程。

sequenceDiagram autonumber participant U as 用户/浏览器 participant FE as 前端
use-document-request participant API as document_api
upload_document participant FS as FileService participant OSS as MinIO participant DB as MySQL U->>FE: 选择文件 FE->>FE: 构造 FormData(file[], parser_config) FE->>API: POST /api/v1/datasets/{id}/documents API->>API: 鉴权 / KB 校验 / 团队权限 / 文件名长度 API->>FS: thread_pool_exec(upload_document) FS->>FS: duplicate_name 处理重名(追加 "_") FS->>OSS: put(kb_id, location, blob) FS->>OSS: put(kb_id, thumbnail_{doc_id}.png, img) OSS-->>FS: ok FS->>DB: DocumentService.insert(document) FS->>DB: add_file_from_kb(file + file2document) DB-->>FS: atomic_increase_doc_num FS-->>API: (doc, blob) API-->>FE: code=0 + 文档元数据 FE-->>U: 列表新增文档(待解析)

5. 异步解析流水线

解析由 parse 接口或勾选 parseOnCreation 触发,经 Redis 队列异步处理。支持 PDF 按页 / Excel 按行并行切分与任务复用。

flowchart TB START["parse 接口 / parseOnCreation"] --> RUN["DocumentService.run"] RUN --> PIPE{"有 pipeline_id?"} PIPE -->|是| DF["queue_dataflow
数据流引擎"] PIPE -->|否| QT["queue_tasks"] QT --> SPLIT{"文件类型"} SPLIT -->|PDF| PG["按 task_page_size 分页"] SPLIT -->|Excel/table| RW["按 3000 行分块"] SPLIT -->|其他| ONE["单任务"] PG --> REUSE RW --> REUSE ONE --> REUSE{"digest 命中旧 chunk?"} REUSE -->|命中| SKIP["复用旧 chunk
删旧索引"] REUSE -->|未命中| INS["bulk_insert task 表"] INS --> MQ[["Redis queue_product"]] MQ --> COL["task_executor.collect
consumer group 拉取"] COL --> BC["build_chunks
get 原文件 → chunker.chunk"] BC --> IMG["chunk 图片回存 MinIO"] BC --> EMBP["embedding 批量向量化
title*w + content*(1-w)"] EMBP --> KB["init_kb: create_idx
ragflow_{tenant_id}"] KB --> INST["docStoreConn.insert
(DOC_BULK_SIZE 批)"] INST --> DONE["increment_chunk_num
更新 progress=1.0"] classDef api fill:#102a44,stroke:#4ea1ff,color:#d6deeb; classDef svc fill:#2a1f3a,stroke:#d2a8ff,color:#d6deeb; classDef store fill:#3a2a18,stroke:#ffab70,color:#d6deeb; class START,RUN,PIPE api; class QT,SPLIT,PG,RW,ONE,REUSE,SKIP,BC,EMBP,KB svc; class INS,MQ,COL,IMG,INST,DONE,DF store;

6. 检索召回链路

前端检索请求经 query 改写、向量编码、混合检索、rerank 到分页返回的完整时序。

sequenceDiagram autonumber participant U as 用户 participant FE as 前端
testing-form/hook participant SVC as knowledge-service participant API as chunk_api
retrieval_test participant EMB as Embedding participant QRY as query.py participant DEAL as Dealer participant DS as 向量库 participant RR as Rerank(可选) U->>FE: question + 阈值/权重/top_k FE->>FE: useWatch 回写表单 FE->>SVC: retrievalTest(params) SVC->>SVC: dataset_id → dataset_ids[] SVC->>API: POST /datasets/search API->>API: 鉴权 + 校验 embedding 一致 API->>EMB: 装载 embedding(必)+rerank(可选) API->>DEAL: retriever.retrieval(...) DEAL->>QRY: question() 改写带权布尔表达式 QRY-->>DEAL: MatchTextExpr + keywords DEAL->>EMB: encode_queries(question) EMB-->>DEAL: MatchDenseExpr(q_dim_vec) DEAL->>DS: search(全文+KNN+Fusion 0.05,0.95) DS-->>DEAL: SearchResult alt 空结果 DEAL->>DS: 降级重试(min_match↓,similarity↑) end alt 有 rerank 模型 DEAL->>RR: similarity(query, docs) RR-->>DEAL: 向量相似度分 else ES 默认 DEAL->>DS: 二次 KNN-only(_knn_scores) DS-->>DEAL: cosine 分 end DEAL->>DEAL: sim=tk*term+vt*vec+rank_feature
阈值过滤+排序+分页 DEAL-->>API: chunks{3类相似度,highlight}+doc_aggs API->>API: 去 vector + 字段名映射 API-->>FE: get_result(data=ranks) FE-->>U: 渲染 chunk/高亮/文档聚合

7. 混合检索内部结构

三段式匹配表达式(全文 + 稠密向量 + 融合)在不同存储后端的差异化处理与最终打分。

flowchart TB Q["question"] --> RW["query.question()
分词+同义词+bigram"] Q --> VE["get_vector
encode_queries"] RW --> MT["MatchTextExpr
字段加权 important_kwd^30..."] VE --> MD["MatchDenseExpr
q_dim_vec cosine"] MT --> FU["FusionExpr
weighted_sum 0.05,0.95"] MD --> FU FU --> ENG{"DOC_ENGINE"} ENG -->|ES| ES["query_string + knn
boost=1-vec_weight
+rank_feature"] ENG -->|Infinity| INF["atan 归一化融合
无需本地 rerank"] ENG -->|OceanBase| OB["回传向量
本地 rerank"] ES --> RK{"重排路径"} INF --> SC1["直接用 _score"] OB --> RK RK -->|有模型| RM["rerank_by_model
外部 similarity"] RK -->|ES默认| KN["_knn_scores
二次KNN取干净cosine"] RM --> FIN KN --> FIN["sim = tk*term + vt*vector + rank_feature"] SC1 --> FIN FIN --> PG["argsort + 阈值过滤 + 分页切片"] PG --> OUT["chunks + doc_aggs"] classDef svc fill:#2a1f3a,stroke:#d2a8ff,color:#d6deeb; classDef store fill:#3a2a18,stroke:#ffab70,color:#d6deeb; classDef api fill:#102a44,stroke:#4ea1ff,color:#d6deeb; class Q,RW,VE,MT,MD,FU,FIN,PG,OUT svc; class ENG,ES,INF,OB store; class RK,RM,KN,SC1 api;
设计要点:融合权重 0.05,0.95(全文:向量)在 ES 端转成 boost = 1 - vector_similarity_weight;ES 路径主检索不回传 chunk 向量,用候选 id 过滤的二次 KNN-only 查询取干净 cosine,再本地与 term 相似度按用户权重加权。

8. 部署 / 容器视图

docker-compose 典型部署:ragflow_server(API+前端静态)、task_executor(解析 worker)与四个有状态中间件。

flowchart TB subgraph HOST["Docker Host / K8s"] subgraph APPC["应用容器"] SRV["ragflow_server
Quart + 静态前端
api/ragflow_server.py"] WORK["task_executor x N
rag/svr/task_executor.py"] end subgraph STATE["有状态服务"] MINIO[("MinIO :9000")] MYSQL[("MySQL :3306")] REDIS[["Redis :6379"]] ESN[("Elasticsearch :9200
/ Infinity")] end end NGINX["Nginx / 入口"] --> SRV SRV --> MINIO SRV --> MYSQL SRV --> REDIS SRV --> ESN WORK --> MINIO WORK --> MYSQL WORK --> REDIS WORK --> ESN SRV -.推任务.-> REDIS REDIS -.拉任务.-> WORK classDef api fill:#102a44,stroke:#4ea1ff,color:#d6deeb; classDef store fill:#3a2a18,stroke:#ffab70,color:#d6deeb; classDef fe fill:#16321f,stroke:#7ee787,color:#d6deeb; class SRV,WORK api; class MINIO,MYSQL,REDIS,ESN store; class NGINX fe;
容器角色入口
ragflow_serverAPI 服务 + 前端静态托管api/ragflow_server.py
task_executor解析/向量化 worker(可横向扩容)rag/svr/task_executor.py
MinIO原始文件 / 缩略图 / chunk 图片bucket = kb_id
MySQL全部关系型元数据Peewee ORM
Redis解析任务队列 + 缓存consumer group
ES / Infinitychunk 混合索引ragflow_{tenant_id}