Skip to main content

StarRocks

StarRocks是一个高性能的分析数据库。 StarRocks是一个面向全面分析场景的下一代亚秒级MPP数据库，包括多维分析、实时分析和即席查询。

通常StarRocks被归类为OLAP，并且在ClickBench - 用于分析DBMS的基准测试中表现出色。由于它具有超快的矢量化执行引擎，它也可以用作快速的矢量数据库。

这里我们将展示如何使用StarRocks Vector Store。

设置​

#!pip install pymysql

在开始时设置update_vectordb = False。如果没有更新文档，则不需要重新构建文档的嵌入

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import StarRocks
from langchain.vectorstores.starrocks import StarRocksSettings
from langchain.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter, TokenTextSplitter
from langchain import OpenAI, VectorDBQA
from langchain.document_loaders import DirectoryLoader
from langchain.chains import RetrievalQA
from langchain.document_loaders import TextLoader, UnstructuredMarkdownLoader

update_vectordb = False
    /Users/dirlt/utils/py3env/lib/python3.9/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.7) or chardet (5.1.0)/charset_normalizer (2.0.9) doesn't match a supported version!
      warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "

加载文档并将其拆分为标记​

加载docs目录下的所有markdown文件

对于starrocks文档，您可以从https://github.com/StarRocks/starrocks克隆存储库，在其中有一个`docs`目录。

loader = DirectoryLoader(
    "./docs", glob="**/*.md", loader_cls=UnstructuredMarkdownLoader
)
documents = loader.load()

将文档拆分为标记，并设置update_vectordb = True，因为有新的文档/标记。

# load text splitter and split docs into snippets of text
text_splitter = TokenTextSplitter(chunk_size=400, chunk_overlap=50)
split_docs = text_splitter.split_documents(documents)

# tell vectordb to update text embeddings
update_vectordb = True
split_docs[-20]
    Document(page_content='Compile StarRocks with Docker\n\nThis topic describes how to compile StarRocks using Docker.\n\nOverview\n\nStarRocks provides development environment images for both Ubuntu 22.04 and CentOS 7.9. With the image, you can launch a Docker container and compile StarRocks in the container.\n\nStarRocks version and DEV ENV image\n\nDifferent branches of StarRocks correspond to different development environment images provided on StarRocks Docker Hub.\n\nFor Ubuntu 22.04:\n\n| Branch name | Image name              |\n  | --------------- | ----------------------------------- |\n  | main            | starrocks/dev-env-ubuntu:latest     |\n  | branch-3.0      | starrocks/dev-env-ubuntu:3.0-latest |\n  | branch-2.5      | starrocks/dev-env-ubuntu:2.5-latest |\n\nFor CentOS 7.9:\n\n| Branch name | Image name                       |\n  | --------------- | ------------------------------------ |\n  | main            | starrocks/dev-env-centos7:latest     |\n  | branch-3.0      | starrocks/dev-env-centos7:3.0-latest |\n  | branch-2.5      | starrocks/dev-env-centos7:2.5-latest |\n\nPrerequisites\n\nBefore compiling StarRocks, make sure the following requirements are satisfied:\n\nHardware\n\n', metadata={'source': 'docs/developers/build-starrocks/Build_in_docker.md'})
print("# docs  = %d, # splits = %d" % (len(documents), len(split_docs)))
    # docs  = 657, # splits = 2802

创建vectordb实例​

使用StarRocks作为vectordb​

def gen_starrocks(update_vectordb, embeddings, settings):
    if update_vectordb:
        docsearch = StarRocks.from_documents(split_docs, embeddings, config=settings)
    else:
        docsearch = StarRocks(embeddings, settings)
    return docsearch

将标记转换为嵌入并将其放入vectordb​

这里我们使用StarRocks作为vectordb，您可以通过StarRocksSettings配置StarRocks实例。

配置StarRocks实例与配置mysql实例非常相似。您需要指定：

  1. 主机/端口
  2. 用户名（默认为'root'）
  3. 密码（默认为空）
  4. 数据库（默认为'default'）
  5. 表（默认为'langchain'）
embeddings = OpenAIEmbeddings()

# configure starrocks settings(host/port/user/pw/db)
settings = StarRocksSettings()
settings.port = 41003
settings.host = "127.0.0.1"
settings.username = "root"
settings.password = ""
settings.database = "zya"
docsearch = gen_starrocks(update_vectordb, embeddings, settings)

print(docsearch)

update_vectordb = False
    Inserting data...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2802/2802 [02:26<00:00, 19.11it/s]


    zya.langchain @ 127.0.0.1:41003
    
    username: root
    
    Table Schema:
    ----------------------------------------------------------------------------
    |name                    |type                    |key                     |
    ----------------------------------------------------------------------------
    |id                      |varchar(65533)          |true                    |
    |document                |varchar(65533)          |false                   |
    |embedding               |array<float>            |false                   |
    |metadata                |varchar(65533)          |false                   |
    ----------------------------------------------------------------------------

构建QA并向其提问​

llm = OpenAI()
qa = RetrievalQA.from_chain_type(
    llm=llm, chain_type="stuff", retriever=docsearch.as_retriever()
)
query = "is profile enabled by default? if not, how to enable profile?"
resp = qa.run(query)
print(resp)
     不，配置文件默认情况下是禁用的。要启用配置文件，请使用命令`set enable_profile = true;`将变量`enable_profile`设置为`true`