Docugami（文档分析）

本文档介绍如何从Docugami加载文档。它提供了使用该系统相比其他数据加载器的优势。

# Docugami
This notebook covers how to load documents from `Docugami`. It provides the advantages of using this system over alternative data loaders.

先决条件 (Prerequisites)

1. 安装必要的Python包 (Install necessary python packages).
2. 获取工作区的访问令牌，并确保将其设置为 DOCUGAMI_API_KEY 环境变量 (Grab an access token for your workspace, and make sure it is set as the DOCUGAMI_API_KEY environment variable).
3. 获取一些已处理文档的文档集和文档ID，如此处所述：https://help.docugami.com/home/docugami-api (Grab some docset and document IDs for your processed documents, as described here: https://help.docugami.com/home/docugami-api).

# 您需要安装lxml包才能使用DocugamiLoader (You need the lxml package to use the DocugamiLoader)
pip install lxml

快速开始 (Quick start)

1. 创建一个Docugami工作空间（提供免费试用）
2. 添加您的文档（PDF、DOCX或DOC），并允许Docugami将它们摄取并聚类成一组相似的文档，例如NDAs、租赁协议和服务协议。系统支持的文档类型没有固定的集合，创建的聚类取决于您的具体文档，并且您可以稍后更改文档集分配。
3. 通过开发者游乐场为您的工作空间创建访问令牌。详细说明
4. 探索Docugami API，以获取已处理的文档集ID列表，或者特定文档集的文档ID列表。
5. 使用下面详细说明的DocugamiLoader，获取文档的丰富语义块。
6. 可选地，构建并发布一个或多个报告或摘要。这有助于Docugami根据您的偏好改进基于语义的XML，并将其作为元数据添加到DocugamiLoader的输出中。使用诸如自查询检索器之类的技术进行高准确度的文档质量保证。

与其他分块技术相比的优势 (Advantages vs Other Chunking Techniques)

适当地对文档进行分块对于从文档中检索信息至关重要。存在许多分块技术，包括依赖于空白字符的简单分块和基于字符长度递归分块拆分。Docugami提供了一种不同的方法：

1. 智能分块： Docugami将每个文档分解为层次化的语义XML树，树中的分块大小各异，从单词或数字值到整个章节。这些分块遵循文档的语义轮廓，提供比任意长度或简单基于空白字符的分块更有意义的表示。
2. 结构化表示： 此外，XML树指示每个文档的结构轮廓，使用属性表示标题、段落、列表、表格和其他常见元素，并在所有支持的文档格式（如扫描的PDF或DOCX文件）中保持一致。它适当处理长篇文档的特征，如页面页眉/页脚或多列流程以进行干净的文本提取。
3. 语义注释： 分块使用语义标签进行注释，这些标签在整个文档集中是连贯的，有助于在多个文档之间进行一致的层次化查询，即使这些文档的撰写和格式不同。例如，在一组租赁协议中，您可以轻松识别关键条款，如房东、租户或续约日期，以及更复杂的信息，例如任何子租赁条款的措辞或特定司法管辖区是否在终止条款中有例外部分。
4. 附加元数据： 如果用户一直在使用Docugami，分块还会带有附加的元数据。这些附加元数据可以用于高准确性的文档质量保证，而无需上下文窗口限制。请参阅下面的详细代码演示。

import os
from langchain.document_loaders import DocugamiLoader

加载文档 (Load Documents)

如果设置了 DOCUGAMI_API_KEY 环境变量，则无需显式地将其传递给加载器；否则，可以将其作为 access_token 参数传递。

DOCUGAMI_API_KEY = os.environ.get("DOCUGAMI_API_KEY")

# 要加载给定文档集 ID 中的所有文档，只需不提供 document_ids 参数
loader = DocugamiLoader(docset_id="ecxqpipcoe2p", document_ids=["43rj0ds7s0ur"])
docs = loader.load()
docs

    [Document(page_content='MUTUAL NON-DISCLOSURE AGREEMENT This  Mutual Non-Disclosure Agreement  (this “ Agreement ”) is entered into and made effective as of  April  4 ,  2018  between  Docugami Inc. , a  Delaware  corporation , whose address is  150  Lake Street South ,  Suite  221 ,  Kirkland ,  Washington  98033 , and  Caleb Divine , an individual, whose address is  1201  Rt  300 ,  Newburgh  NY  12550 .', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:ThisMutualNon-disclosureAgreement', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'ThisMutualNon-disclosureAgreement'}),
     Document(page_content='The above named parties desire to engage in discussions regarding a potential agreement or other transaction between the parties (the “Purpose”). In connection with such discussions, it may be necessary for the parties to disclose to each other certain confidential information or materials to enable them to evaluate whether to enter into such agreement or transaction.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Discussions', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'Discussions'}),
     Document(page_content='In consideration of the foregoing, the parties agree as follows:', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Consideration', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'Consideration'}),
     Document(page_content='1. Confidential Information . For purposes of this  Agreement , “ Confidential Information ” means any information or materials disclosed by  one  party  to the other party that: (i) if disclosed in writing or in the form of tangible materials, is marked “confidential” or “proprietary” at the time of such disclosure; (ii) if disclosed orally or by visual presentation, is identified as “confidential” or “proprietary” at the time of such disclosure, and is summarized in a writing sent by the disclosing party to the receiving party within  thirty  ( 30 ) days  after any such disclosure; or (iii) due to its nature or the circumstances of its disclosure, a person exercising reasonable business judgment would understand to be confidential or proprietary.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:Purposes/docset:ConfidentialInformation-section/docset:ConfidentialInformation[2]', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'ConfidentialInformation'}),
     Document(page_content="2. Obligations and  Restrictions . Each party agrees: (i) to maintain the  other party's Confidential Information  in strict confidence; (ii) not to disclose  such Confidential Information  to any third party; and (iii) not to use  such Confidential Information  for any purpose except for the Purpose. Each party may disclose the  other party’s Confidential Information  to its employees and consultants who have a bona fide need to know  such Confidential Information  for the Purpose, but solely to the extent necessary to pursue the  Purpose  and for no other purpose; provided, that each such employee and consultant first executes a written agreement (or is otherwise already bound by a written agreement) that contains use and nondisclosure restrictions at least as protective of the  other party’s Confidential Information  as those set forth in this  Agreement .", metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:Obligations/docset:ObligationsAndRestrictions-section/docset:ObligationsAndRestrictions', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'ObligationsAndRestrictions'}),
     Document(page_content='3. Exceptions. The obligations and restrictions in Section  2  will not apply to any information or materials that:', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:Exceptions/docset:Exceptions-section/docset:Exceptions[2]', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'Exceptions'}),
     Document(page_content='(i) were, at the date of disclosure, or have subsequently become, generally known or available to the public through no act or failure to act by the receiving party;', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:TheDate/docset:TheDate/docset:TheDate', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'TheDate'}),
     Document(page_content='(ii) were rightfully known by the receiving party prior to receiving such information or materials from the disclosing party;', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:TheDate/docset:SuchInformation/docset:TheReceivingParty', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'TheReceivingParty'}),
     Document(page_content='(iii) are rightfully acquired by the receiving party from a third party who has the right to disclose such information or materials without breach of any confidentiality obligation to the disclosing party;', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:TheDate/docset:TheReceivingParty/docset:TheReceivingParty', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'TheReceivingParty'}),
     Document(page_content='4. Compelled Disclosure . Nothing in this  Agreement  will be deemed to restrict a party from disclosing the  other party’s Confidential Information  to the extent required by any order, subpoena, law, statute or regulation; provided, that the party required to make such a disclosure uses reasonable efforts to give the other party reasonable advance notice of such required disclosure in order to enable the other party to prevent or limit such disclosure.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:Disclosure/docset:CompelledDisclosure-section/docset:CompelledDisclosure', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'CompelledDisclosure'}),
     Document(page_content='5. Return of  Confidential Information . Upon the completion or abandonment of the Purpose, and in any event upon the disclosing party’s request, the receiving party will promptly return to the disclosing party all tangible items and embodiments containing or consisting of the  disclosing party’s Confidential Information  and all copies thereof (including electronic copies), and any notes, analyses, compilations, studies, interpretations, memoranda or other documents (regardless of the form thereof) prepared by or on behalf of the receiving party that contain or are based upon the  disclosing party’s Confidential Information .', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:TheCompletion/docset:ReturnofConfidentialInformation-section/docset:ReturnofConfidentialInformation', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'ReturnofConfidentialInformation'}),
     Document(page_content='6. No  Obligations . Each party retains the right to determine whether to disclose any  Confidential Information  to the other party.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:NoObligations/docset:NoObligations-section/docset:NoObligations[2]', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'NoObligations'}),
     Document(page_content='7. No Warranty. ALL  CONFIDENTIAL INFORMATION  IS PROVIDED BY THE  DISCLOSING PARTY  “AS  IS ”.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:NoWarranty/docset:NoWarranty-section/docset:NoWarranty[2]', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'NoWarranty'}),
     Document(page_content='8. Term. This  Agreement  will remain in effect for a period of  seven  ( 7 ) years  from the date of last disclosure of  Confidential Information  by either party, at which time it will terminate.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:ThisAgreement/docset:Term-section/docset:Term', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'Term'}),
     Document(page_content='9. Equitable Relief . Each party acknowledges that the unauthorized use or disclosure of the  disclosing party’s Confidential Information  may cause the disclosing party to incur irreparable harm and significant damages, the degree of which may be difficult to ascertain. Accordingly, each party agrees that the disclosing party will have the right to seek immediate equitable relief to enjoin any unauthorized use or disclosure of  its Confidential Information , in addition to any other rights and remedies that it may have at law or otherwise.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:EquitableRelief/docset:EquitableRelief-section/docset:EquitableRelief[2]', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'EquitableRelief'}),
     Document(page_content='10. Non-compete. To the maximum extent permitted by applicable law, during the  Term  of this  Agreement  and for a period of  one  ( 1 ) year  thereafter,  Caleb  Divine  may not market software products or do business that directly or indirectly competes with  Docugami  software products .', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:TheMaximumExtent/docset:Non-compete-section/docset:Non-compete', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'Non-compete'}),
     Document(page_content='11. Miscellaneous. This  Agreement  will be governed and construed in accordance with the laws of the  State  of  Washington , excluding its body of law controlling conflict of laws. This  Agreement  is the complete and exclusive understanding and agreement between the parties regarding the subject matter of this  Agreement  and supersedes all prior agreements, understandings and communications, oral or written, between the parties regarding the subject matter of this  Agreement . If any provision of this  Agreement  is held invalid or unenforceable by a court of competent jurisdiction, that provision of this  Agreement  will be enforced to the maximum extent permissible and the other provisions of this  Agreement  will remain in full force and effect. Neither party may assign this  Agreement , in whole or in part, by operation of law or otherwise, without the other party’s prior written consent, and any attempted assignment without such consent will be void. This  Agreement  may be executed in counterparts, each of which will be deemed an original, but all of which together will constitute one and the same instrument.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:Accordance/docset:Miscellaneous-section/docset:Miscellaneous', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'Miscellaneous'}),
     Document(page_content='[SIGNATURE PAGE FOLLOWS] IN  WITNESS  WHEREOF, the parties hereto have executed this  Mutual Non-Disclosure Agreement  by their duly authorized officers or representatives as of the date first set forth above.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:Witness/docset:TheParties/docset:TheParties', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'TheParties'}),
     Document(page_content='DOCUGAMI INC . : \n\n Caleb Divine : \n\n Signature:  Signature:  Name: \n\n Jean Paoli  Name:  Title: \n\n CEO  Title:', metadata={'xpath': '/docset:MutualNon-disclosure/docset:Witness/docset:TheParties/docset:DocugamiInc/docset:DocugamiInc/xhtml:table', 'id': '43rj0ds7s0ur', 'name': 'NDA simple layout.docx', 'structure': '', 'tag': 'table'})]

每个 Document 的 metadata (实际上是 PDF、DOC 或 DOCX 的一部分) 包含一些有用的附加信息:

1. id 和 name: 文件 (PDF、DOC 或 DOCX) 的 ID 和名称，该块来源于 Docugami 中的文件。
2. xpath: 块在文档的 XML 表示中的 XPath。对于直接引用文档 XML 中的实际块的源引用很有用。
3. structure: 块的结构属性，例如 h1、h2、div、table、td 等。如果需要，对于过滤掉某些类型的块很有用。
4. tag: 使用各种生成和提取技术的块的语义标签。更多详细信息请参阅：https://github.com/docugami/DFM-benchmarks

基本用法：用于文档质量分析的Docugami加载器

您可以像使用标准加载器一样使用Docugami加载器进行多个文档的文档质量分析，只是使用的块更好地遵循文档的自然轮廓。有很多关于如何做到这一点的优秀教程，例如这个。我们可以使用相同的代码，但是使用DocugamiLoader进行更好的分块，而不是直接使用基本的拆分技术加载文本或PDF文件。

poetry run pip -q install openai tiktoken chromadb

from langchain.schema import Document
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.llms import OpenAI
from langchain.chains import RetrievalQA

# 对于这个示例，我们已经有了一组租赁文件的处理过的文档集
loader = DocugamiLoader(docset_id="wh2kned25uqm")
documents = loader.load()

加载器返回的文档已经被拆分，因此我们不需要使用文本拆分器。可选地，我们可以使用每个文档上的元数据，例如结构或标签属性，来进行任何我们想要的后处理。

我们将直接使用DocugamiLoader的输出来设置一个常规的检索QA链。

embedding = OpenAIEmbeddings()
vectordb = Chroma.from_documents(documents=documents, embedding=embedding)
retriever = vectordb.as_retriever()
qa_chain = RetrievalQA.from_chain_type(
    llm=OpenAI(), chain_type="stuff", retriever=retriever, return_source_documents=True
)

    Using embedded DuckDB without persistence: data will be transient

# 使用示例查询测试检索器
qa_chain("What can tenants do with signage on their properties?")

    {'query': 'What can tenants do with signage on their properties?',
     'result': ' Tenants may place signs (digital or otherwise) or other form of identification on the premises after receiving written permission from the landlord which shall not be unreasonably withheld. The tenant is responsible for any damage caused to the premises and must conform to any applicable laws, ordinances, etc. governing the same. The tenant must also remove and clean any window or glass identification promptly upon vacating the premises.',
     'source_documents': [Document(page_content='ARTICLE VI  SIGNAGE 6.01  Signage . Tenant  may place or attach to the  Premises signs  (digital or otherwise) or other such identification as needed after receiving written permission from the  Landlord , which permission shall not be unreasonably withheld. Any damage caused to the Premises by the  Tenant ’s erecting or removing such signs shall be repaired promptly by the  Tenant  at the  Tenant ’s expense . Any signs or other form of identification allowed must conform to all applicable laws, ordinances, etc. governing the same.  Tenant  also agrees to have any window or glass identification completely removed and cleaned at its expense promptly upon vacating the Premises.', metadata={'xpath': '/docset:OFFICELEASEAGREEMENT-section/docset:OFFICELEASEAGREEMENT/docset:Article/docset:ARTICLEVISIGNAGE-section/docset:_601Signage-section/docset:_601Signage', 'id': 'v1bvgaozfkak', 'name': 'TruTone Lane 2.docx', 'structure': 'div', 'tag': '_601Signage', 'Landlord': 'BUBBA CENTER PARTNERSHIP', 'Tenant': 'Truetone Lane LLC'}),
      Document(page_content='Signage.  Tenant  may place or attach to the  Premises signs  (digital or otherwise) or other such identification as needed after receiving written permission from the  Landlord , which permission shall not be unreasonably withheld. Any damage caused to the Premises by the  Tenant ’s erecting or removing such signs shall be repaired promptly by the  Tenant  at the  Tenant ’s expense . Any signs or other form of identification allowed must conform to all applicable laws, ordinances, etc. governing the same.  Tenant  also agrees to have any window or glass identification completely removed and cleaned at its expense promptly upon vacating the Premises. \n\n                                                          ARTICLE  VII  UTILITIES 7.01', metadata={'xpath': '/docset:OFFICELEASEAGREEMENT-section/docset:OFFICELEASEAGREEMENT/docset:ThisOFFICELEASEAGREEMENTThis/docset:ArticleIBasic/docset:ArticleIiiUseAndCareOf/docset:ARTICLEIIIUSEANDCAREOFPREMISES-section/docset:ARTICLEIIIUSEANDCAREOFPREMISES/docset:NoOtherPurposes/docset:TenantsResponsibility/dg:chunk', 'id': 'g2fvhekmltza', 'name': 'TruTone Lane 6.pdf', 'structure': 'lim', 'tag': 'chunk', 'Landlord': 'GLORY ROAD LLC', 'Tenant': 'Truetone Lane LLC'}),
      Document(page_content='Landlord , its agents, servants, employees, licensees, invitees, and contractors during the last year of the term of this  Lease  at any and all times during regular business hours, after  24  hour  notice  to tenant, to pass and repass on and through the Premises, or such portion thereof as may be necessary, in order that they or any of them may gain access to the Premises for the purpose of showing the  Premises  to potential new tenants or real estate brokers. In addition,  Landlord  shall be entitled to place a "FOR  RENT " or "FOR LEASE" sign (not exceeding  8.5 ” x  11 ”) in the front window of the Premises during the  last  six  months  of the term of this  Lease .', metadata={'xpath': '/docset:Rider/docset:RIDERTOLEASE-section/docset:RIDERTOLEASE/docset:FixedRent/docset:TermYearPeriod/docset:Lease/docset:_42FLandlordSAccess-section/docset:_42FLandlordSAccess/docset:LandlordsRights/docset:Landlord', 'id': 'omvs4mysdk6b', 'name': 'TruTone Lane 1.docx', 'structure': 'p', 'tag': 'Landlord', 'Landlord': 'BIRCH STREET ,  LLC', 'Tenant': 'Trutone Lane LLC'}),
      Document(page_content="24. SIGNS . No signage shall be placed by  Tenant  on any portion of the  Project . However,  Tenant  shall be permitted to place a sign bearing its name in a location approved by  Landlord  near the entrance to the  Premises  (at  Tenant's cost ) and will be furnished a single listing of its name in the  Building's directory  (at  Landlord 's cost ), all in accordance with the criteria adopted  from time to time  by  Landlord  for the  Project . Any changes or additional listings in the directory shall be furnished (subject to availability of space) for the  then Building Standard charge .", metadata={'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:GrossRentCreditTheRentCredit-section/docset:GrossRentCreditTheRentCredit/docset:Period/docset:ApplicableSalesTax/docset:PercentageRent/docset:TheTerms/docset:Indemnification/docset:INDEMNIFICATION-section/docset:INDEMNIFICATION/docset:Waiver/docset:Waiver/docset:Signs/docset:SIGNS-section/docset:SIGNS', 'id': 'qkn9cyqsiuch', 'name': 'Shorebucks LLC_AZ.pdf', 'structure': 'div', 'tag': 'SIGNS', 'Landlord': 'Menlo Group', 'Tenant': 'Shorebucks LLC'})]}

使用Docugami为高准确度文档QA添加元数据

大型文档的一个问题是，对于您的问题的正确答案可能取决于文档中相距较远的块。即使使用重叠的典型分块技术，也很难提供LLM足够的上下文来回答这些问题。随着即将推出的非常大的上下文LLM，可能可以将大量的令牌，甚至整个文档，放入上下文中，但是对于非常长的文档或大量文档，这仍然会达到某个限制。

例如，如果我们提出一个更复杂的问题，需要LLM从文档的不同部分获取块，即使是OpenAI强大的LLM也无法正确回答。

chain_response = qa_chain("What is rentable area for the property owned by DHA Group?")
chain_response["result"]  # 正确答案应该是13,500

    ' 9,753 square feet'

乍一看，答案似乎是合理的，但是如果您仔细查看此答案的源块，您会发现文档的分块并没有将房东姓名和可出租面积放在同一上下文中，因为它们在文档中相距较远。因此，检索器最终会找到来自其他文档的不相关块，这些文档甚至与Menlo Group房东无关。该房东恰好在文件Shorebucks LLC_NJ.pdf的第一页中提到，虽然链中使用的一个源块确实来自包含正确答案（13,500）的该文档，但还包括来自不同文档的其他源块，因此答案是错误的。

chain_response["source_documents"]

    [Document(page_content='1.1 Landlord . DHA Group , a  Delaware  limited liability company  authorized to transact business in  New Jersey .', metadata={'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:TheTerms/dg:chunk/docset:BasicLeaseInformation/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS-section/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS/docset:DhaGroup/docset:DhaGroup/docset:DhaGroup/docset:Landlord-section/docset:DhaGroup', 'id': 'md8rieecquyv', 'name': 'Shorebucks LLC_NJ.pdf', 'structure': 'div', 'tag': 'DhaGroup', 'Landlord': 'DHA Group', 'Tenant': 'Shorebucks LLC'}),
     Document(page_content='WITNESSES: LANDLORD: DHA Group , a  Delaware  limited liability company', metadata={'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:GrossRentCreditTheRentCredit-section/docset:GrossRentCreditTheRentCredit/docset:Guaranty-section/docset:Guaranty[2]/docset:SIGNATURESONNEXTPAGE-section/docset:INWITNESSWHEREOF-section/docset:INWITNESSWHEREOF/docset:Behalf/docset:Witnesses/xhtml:table/xhtml:tbody/xhtml:tr[3]/xhtml:td[2]/docset:DhaGroup', 'id': 'md8rieecquyv', 'name': 'Shorebucks LLC_NJ.pdf', 'structure': 'p', 'tag': 'DhaGroup', 'Landlord': 'DHA Group', 'Tenant': 'Shorebucks LLC'}),
     Document(page_content="1.16 Landlord 's Notice Address . DHA  Group , Suite  1010 ,  111  Bauer Dr ,  Oakland ,  New Jersey ,  07436 , with a copy to the  Building  Management  Office  at the  Project , Attention:  On - Site  Property Manager .", metadata={'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:GrossRentCreditTheRentCredit-section/docset:GrossRentCreditTheRentCredit/docset:Period/docset:ApplicableSalesTax/docset:PercentageRent/docset:PercentageRent/docset:NoticeAddress[2]/docset:LandlordsNoticeAddress-section/docset:LandlordsNoticeAddress[2]', 'id': 'md8rieecquyv', 'name': 'Shorebucks LLC_NJ.pdf', 'structure': 'div', 'tag': 'LandlordsNoticeAddress', 'Landlord': 'DHA Group', 'Tenant': 'Shorebucks LLC'}),
     Document(page_content='1.6 Rentable Area  of the Premises. 9,753  square feet . This square footage figure includes an add-on factor for  Common Areas  in the Building and has been agreed upon by the parties as final and correct and is not subject to challenge or dispute by either party.', metadata={'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:TheTerms/dg:chunk/docset:BasicLeaseInformation/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS-section/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS/docset:PerryBlair/docset:PerryBlair/docset:Premises[2]/docset:RentableAreaofthePremises-section/docset:RentableAreaofthePremises', 'id': 'dsyfhh4vpeyf', 'name': 'Shorebucks LLC_CO.pdf', 'structure': 'div', 'tag': 'RentableAreaofthePremises', 'Landlord': 'Perry  &  Blair LLC', 'Tenant': 'Shorebucks LLC'})]

在这里，Docugami可以提供帮助。如果用户一直在使用Docugami，块将使用不同的技术进行注释，并创建附加的元数据。稍后将添加更多技术性的方法。

具体来说，让我们看一下由docugami返回的文档上的附加元数据，以简单的键/值对的形式：

loader = DocugamiLoader(docset_id="wh2kned25uqm")
documents = loader.load()
documents[0].metadata

    {'xpath': '/docset:OFFICELEASEAGREEMENT-section/docset:OFFICELEASEAGREEMENT/docset:ThisOfficeLeaseAgreement',
     'id': 'v1bvgaozfkak',
     'name': 'TruTone Lane 2.docx',
     'structure': 'p',
     'tag': 'ThisOfficeLeaseAgreement',
     'Landlord': 'BUBBA CENTER PARTNERSHIP',
     'Tenant': 'Truetone Lane LLC'}

我们可以使用自查询检索器来提高查询准确性，使用这些附加的元数据：

from langchain.chains.query_constructor.schema import AttributeInfo
from langchain.retrievers.self_query.base import SelfQueryRetriever

EXCLUDE_KEYS = ["id", "xpath", "structure"]
metadata_field_info = [
    AttributeInfo(
        name=key,
        description=f"The {key} for this chunk",
        type="string",
    )
    for key in documents[0].metadata
    if key.lower() not in EXCLUDE_KEYS
]


document_content_description = "Contents of this chunk"
llm = OpenAI(temperature=0)
vectordb = Chroma.from_documents(documents=documents, embedding=embedding)
retriever = SelfQueryRetriever.from_llm(
    llm, vectordb, document_content_description, metadata_field_info, verbose=True
)
qa_chain = RetrievalQA.from_chain_type(
    llm=OpenAI(), chain_type="stuff", retriever=retriever, return_source_documents=True
)

    Using embedded DuckDB without persistence: data will be transient

让我们再次运行相同的问题。由于所有块都具有元数据键/值对，这些键/值对携带有关文档的关键信息，即使这些信息在物理上与用于生成答案的特定块非常远。

qa_chain("What is rentable area for the property owned by DHA Group?")

    query='rentable area' filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='Landlord', value='DHA Group')





    {'query': 'What is rentable area for the property owned by DHA Group?',
     'result': ' 13,500 square feet.',
     'source_documents': [Document(page_content='1.1 Landlord . DHA Group , a  Delaware  limited liability company  authorized to transact business in  New Jersey .', metadata={'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:TheTerms/dg:chunk/docset:BasicLeaseInformation/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS-section/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS/docset:DhaGroup/docset:DhaGroup/docset:DhaGroup/docset:Landlord-section/docset:DhaGroup', 'id': 'md8rieecquyv', 'name': 'Shorebucks LLC_NJ.pdf', 'structure': 'div', 'tag': 'DhaGroup', 'Landlord': 'DHA Group', 'Tenant': 'Shorebucks LLC'}),
      Document(page_content='WITNESSES: LANDLORD: DHA Group , a  Delaware  limited liability company', metadata={'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:GrossRentCreditTheRentCredit-section/docset:GrossRentCreditTheRentCredit/docset:Guaranty-section/docset:Guaranty[2]/docset:SIGNATURESONNEXTPAGE-section/docset:INWITNESSWHEREOF-section/docset:INWITNESSWHEREOF/docset:Behalf/docset:Witnesses/xhtml:table/xhtml:tbody/xhtml:tr[3]/xhtml:td[2]/docset:DhaGroup', 'id': 'md8rieecquyv', 'name': 'Shorebucks LLC_NJ.pdf', 'structure': 'p', 'tag': 'DhaGroup', 'Landlord': 'DHA Group', 'Tenant': 'Shorebucks LLC'}),
      Document(page_content="1.16 Landlord 's Notice Address . DHA  Group , Suite  1010 ,  111  Bauer Dr ,  Oakland ,  New Jersey ,  07436 , with a copy to the  Building  Management  Office  at the  Project , Attention:  On - Site  Property Manager .", metadata={'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:GrossRentCreditTheRentCredit-section/docset:GrossRentCreditTheRentCredit/docset:Period/docset:ApplicableSalesTax/docset:PercentageRent/docset:PercentageRent/docset:NoticeAddress[2]/docset:LandlordsNoticeAddress-section/docset:LandlordsNoticeAddress[2]', 'id': 'md8rieecquyv', 'name': 'Shorebucks LLC_NJ.pdf', 'structure': 'div', 'tag': 'LandlordsNoticeAddress', 'Landlord': 'DHA Group', 'Tenant': 'Shorebucks LLC'}),
      Document(page_content='1.6 Rentable Area  of the Premises. 13,500  square feet . This square footage figure includes an add-on factor for  Common Areas  in the Building and has been agreed upon by the parties as final and correct and is not subject to challenge or dispute by either party.', metadata={'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:TheTerms/dg:chunk/docset:BasicLeaseInformation/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS-section/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS/docset:DhaGroup/docset:DhaGroup/docset:Premises[2]/docset:RentableAreaofthePremises-section/docset:RentableAreaofthePremises', 'id': 'md8rieecquyv', 'name': 'Shorebucks LLC_NJ.pdf', 'structure': 'div', 'tag': 'RentableAreaofthePremises', 'Landlord': 'DHA Group', 'Tenant': 'Shorebucks LLC'})]}

这次答案是正确的，因为自查询检索器在元数据的房东属性上创建了一个过滤器，正确地过滤到了特定的文档，该文档专门涉及DHA Group房东。生成的源块都与该房东相关，即使在包含正确答案的特定块中没有直接提到该房东，这也提高了答案的准确性。