因果程序辅助语言（CPAL）链 Causal program-aided language (CPAL) chain
CPAL链基于最近的PAL来阻止LLM的幻觉。PAL方法的问题在于它在具有嵌套依赖链的数学问题上产生了幻觉。这里的创新之处在于这种新的CPAL方法包括因果结构来修复幻觉。
原始的PR描述包含了完整的概述。
使用CPAL链，LLM将这个
"Tim buys the same number of pets as Cindy and Boris."
"Cindy buys the same number of pets as Bill plus Bob."
"Boris buys the same number of pets as Ben plus Beth."
"Bill buys the same number of pets as Obama."
"Bob buys the same number of pets as Obama."
"Ben buys the same number of pets as Obama."
"Beth buys the same number of pets as Obama."
"If Obama buys one pet, how many pets total does everyone buy?"
转化为这个
.
在这个笔记本中演示的代码示例概述。
- CPAL对幻觉的价值：CPAL vs PAL 1.1 复杂叙述 1.2 无法回答的数学问题
- CPAL的三种因果图类型（来自《为什么的书》）。 2.1 中介者 2.2 碰撞者 2.3 混淆因素
from IPython.display import SVG
from langchain_experimental.cpal.base import CPALChain
from langchain_experimental.pal_chain import PALChain
from langchain import OpenAI
llm = OpenAI(temperature=0, max_tokens=512)
cpal_chain = CPALChain.from_univariate_prompt(llm=llm, verbose=True)
pal_chain = PALChain.from_math_prompt(llm=llm, verbose=True)
CPAL对抗幻觉的价值：CPAL vs PAL
与PAL一样，CPAL旨在减少大型语言模型（LLM）的幻觉。
CPAL链与PAL链有几个不同之处。
CPAL添加了一个因果结构（或DAG）来链接实体操作（或数学表达式）。
CPAL数学表达式模拟了一系列因果关系，可以进行干预，而PAL链的数学表达式是投影的数学身份。
1.1 复杂的叙述 Complex narrative
要点：PAL产生幻觉，CPAL不产生幻觉。
question = (
"Tim buys the same number of pets as Cindy and Boris."
"Cindy buys the same number of pets as Bill plus Bob."
"Boris buys the same number of pets as Ben plus Beth."
"Bill buys the same number of pets as Obama."
"Bob buys the same number of pets as Obama."
"Ben buys the same number of pets as Obama."
"Beth buys the same number of pets as Obama."
"If Obama buys one pet, how many pets total does everyone buy?"
)
pal_chain.run(question)
> Entering new chain...
def solution():
"""Tim buys the same number of pets as Cindy and Boris.Cindy buys the same number of pets as Bill plus Bob.Boris buys the same number of pets as Ben plus Beth.Bill buys the same number of pets as Obama.Bob buys the same number of pets as Obama.Ben buys the same number of pets as Obama.Beth buys the same number of pets as Obama.If Obama buys one pet, how many pets total does everyone buy?"""
obama_pets = 1
tim_pets = obama_pets
cindy_pets = obama_pets + obama_pets
boris_pets = obama_pets + obama_pets
total_pets = tim_pets + cindy_pets + boris_pets
result = total_pets
return result
> Finished chain.
'5'
cpal_chain.run(question)
> Entering new chain...
story outcome data
name code value depends_on
0 obama pass 1.0 []
1 bill bill.value = obama.value 1.0 [obama]
2 bob bob.value = obama.value 1.0 [obama]
3 ben ben.value = obama.value 1.0 [obama]
4 beth beth.value = obama.value 1.0 [obama]
5 cindy cindy.value = bill.value + bob.value 2.0 [bill, bob]
6 boris boris.value = ben.value + beth.value 2.0 [ben, beth]
7 tim tim.value = cindy.value + boris.value 4.0 [cindy, boris]
query data
{
"question": "how many pets total does everyone buy?",
"expression": "SELECT SUM(value) FROM df",
"llm_error_msg": ""
}
> Finished chain.
13.0
# wait 20 secs to see display
cpal_chain.draw(path="web.svg")
SVG("web.svg")
无法回答的数学问题 Unanswerable math
要点：PAL产生幻觉，而CPAL不产生幻觉，而是回答“无法回答，叙述问题和情节不连贯”
question = (
"Jan has three times the number of pets as Marcia."
"Marcia has two more pets than Cindy."
"If Cindy has ten pets, how many pets does Barak have?"
)
pal_chain.run(question)
> Entering new chain...
def solution():
"""Jan has three times the number of pets as Marcia.Marcia has two more pets than Cindy.If Cindy has ten pets, how many pets does Barak have?"""
cindy_pets = 10
marcia_pets = cindy_pets + 2
jan_pets = marcia_pets * 3
result = jan_pets
return result
> Finished chain.
'36'
try:
cpal_chain.run(question)
except Exception as e_msg:
print(e_msg)
> Entering new chain...
story outcome data
name code value depends_on
0 cindy pass 10.0 []
1 marcia marcia.value = cindy.value + 2 12.0 [cindy]
2 jan jan.value = marcia.value * 3 36.0 [marcia]
query data
{
"question": "how many pets does barak have?",
"expression": "SELECT name, value FROM df WHERE name = 'barak'",
"llm_error_msg": ""
}
unanswerable, query and outcome are incoherent
outcome:
name code value depends_on
0 cindy pass 10.0 []
1 marcia marcia.value = cindy.value + 2 12.0 [cindy]
2 jan jan.value = marcia.value * 3 36.0 [marcia]
query:
{'question': 'how many pets does barak have?', 'expression': "SELECT name, value FROM df WHERE name = 'barak'", 'llm_error_msg': ''}
基本数学 Basic math
因果中介 Causal mediator
question = (
"Jan has three times the number of pets as Marcia. "
"Marcia has two more pets than Cindy. "
"If Cindy has four pets, how many total pets do the three have?"
)
PAL
pal_chain.run(question)
> Entering new chain...
def solution():
"""Jan has three times the number of pets as Marcia. Marcia has two more pets than Cindy. If Cindy has four pets, how many total pets do the three have?"""
cindy_pets = 4
marcia_pets = cindy_pets + 2
jan_pets = marcia_pets * 3
total_pets = cindy_pets + marcia_pets + jan_pets
result = total_pets
return result
> Finished chain.
'28'
CPAL
cpal_chain.run(question)
> Entering new chain...
story outcome data
name code value depends_on
0 cindy pass 4.0 []
1 marcia marcia.value = cindy.value + 2 6.0 [cindy]
2 jan jan.value = marcia.value * 3 18.0 [marcia]
query data
{
"question": "how many total pets do the three have?",
"expression": "SELECT SUM(value) FROM df",
"llm_error_msg": ""
}
> Finished chain.
28.0
# wait 20 secs to see display
cpal_chain.draw(path="web.svg")
SVG("web.svg")
因果碰撞 Causal collider
question = (
"Jan has the number of pets as Marcia plus the number of pets as Cindy. "
"Marcia has no pets. "
"If Cindy has four pets, how many total pets do the three have?"
)
cpal_chain.run(question)
> Entering new chain...
story outcome data
name code value depends_on
0 marcia pass 0.0 []
1 cindy pass 4.0 []
2 jan jan.value = marcia.value + cindy.value 4.0 [marcia, cindy]
query data
{
"question": "how many total pets do the three have?",
"expression": "SELECT SUM(value) FROM df",
"llm_error_msg": ""
}
> Finished chain.
8.0
# wait 20 secs to see display
cpal_chain.draw(path="web.svg")
SVG("web.svg")
因果混淆 Causal confounder
question = (
"Jan has the number of pets as Marcia plus the number of pets as Cindy. "
"Marcia has two more pets than Cindy. "
"If Cindy has four pets, how many total pets do the three have?"
)
cpal_chain.run(question)
> Entering new chain...
story outcome data
name code value depends_on
0 cindy pass 4.0 []
1 marcia marcia.value = cindy.value + 2 6.0 [cindy]
2 jan jan.value = cindy.value + marcia.value 10.0 [cindy, marcia]
query data
{
"question": "how many total pets do the three have?",
"expression": "SELECT SUM(value) FROM df",
"llm_error_msg": ""
}
> Finished chain.
20.0
# wait 20 secs to see display
cpal_chain.draw(path="web.svg")
SVG("web.svg")
%autoreload 2
