递归按字符分割

这个文本分割器是用于通用文本的推荐分割器。它通过一个字符列表进行参数化。它尝试按顺序在这些字符上进行分割，直到块的大小足够小。默认的字符列表是["\n\n", "\n", " ", ""]。这样做的效果是尽可能地保持段落（然后是句子，然后是单词）在一起，因为它们通常是语义相关的文本片段。

1. 文本如何分割：通过字符列表
2. 块大小如何衡量：通过字符数

# 这是一个我们可以分割的长文档。
with open('../../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    # 设置一个非常小的块大小，只是为了展示。
    chunk_size=100,
    chunk_overlap=20,
    length_function=len,
    is_separator_regex=False,
)

texts = text_splitter.create_documents([state_of_the_union])
print(texts[0])
print(texts[1])

page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and' lookup_str='' metadata={} lookup_index=0
page_content='of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.' lookup_str='' metadata={} lookup_index=0

text_splitter.split_text(state_of_the_union)[:2]

['Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and',
 'of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.']