Chapter 3 Logic Bot solution still gives false conclusions after proposed "fix"

Open
#34 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
ai, documentation

Research direction

Start with the Chapter 3 logic-bot exercise and its system and user prompt examples. Review the reported response against the stated looking relationships and unknown marital status; done means the example no longer presents an unsupported Jack-to-George relationship as a valid conclusion.

Written by the indexing model from the issue text.

Description

Current proposed solution is:

# System prompt
SYSTEM_PROMPT = "You are a logic bot designed to answer complex logic problems."

# Prompt
PROMPT = "Jack is looking at Anne. Anne is looking at George. Jack is married, George is not, and we don’t know if Anne is married. Is a married person looking at an unmarried person?"

# Print Claude's response
print(get_completion(PROMPT, SYSTEM_PROMPT))

Response Analysis

The final response is YES, that is correct, however the explanation contains false conclusions.
The answer it gives, in summary, is, with my comment between square brackets

[...]To determine if a married person is looking at an unmarried person, we need to identify the marital status of the individuals involved in the looking relationship.

From the given information, we know that:
- Jack is married. [true from the prompt]
- George is not married. [true from the prompt]

Therefore, a married person (Jack) is looking at an unmarried person (George). [false, Jack may no be looking at George]

The fact that we don't know Anne's marital status does not affect the conclusion, as the question is asking about the relationship between a married person and an unmarried person, which is satisfied by the relationship between Jack and George. [false, there is no relationship between Jack and George]

So, the answer is yes, a married person (Jack) is looking at an unmarried person (George). [false, if it had removed the names it would be correct]

However "looking" is not a transitive relationship, so we cannot conclude that Jack is looking at George.
Think of a corner in a street with a building in it, Jack may be in street A, George in street B and Anne in that corner between streets A and B, being visible to Jack and being able to see George, but, because of the building, Jack cannot see George.

Attempts to get a correct result

I've tried to fix it on a few ways with different results, but no success.

  1. "Looking at": I've tried to append a reminder at the end of the user prompt: (note that "looking at" is not a transitive relationship)
  • Final System prompt: "You are a logic bot designed to answer complex logic problems."
  • Final User prompt: "User: Jack is looking at Anne. Anne is looking at George. Jack is married, George is not, and we don’t know if Anne is married. Is a married person looking at an unmarried person? (note that "looking at" is not a transitive relationship)"
  1. Asking the system to use a truth table: I've tried appending a solution method to the role "You like to use a truth table to examine the alternatives" so it would try all possibilities.
  • Final System prompt: "You are a logic bot designed to answer complex logic problems. You like to use a truth table to examine the alternatives."
  • Final User prompt: "User: Jack is looking at Anne. Anne is looking at George. Jack is married, George is not, and we don’t know if Anne is married. Is a married person looking at an unmarried person? (note that "looking at" is not a transitive relationship)"
  1. Removing impossible situations: I've tried appending a solution method to the role "You only add the possible possibilities, being able to remove impossible situations"
  • Final System prompt: "You are a logic bot designed to answer complex logic problems. You like to use a truth table to examine the alternatives. You only add the possible possibilities, being able to remove impossible situations."
  • Final User prompt: "User: Jack is looking at Anne. Anne is looking at George. Jack is married, George is not, and we don’t know if Anne is married. Is a married person looking at an unmarried person? (note that "looking at" is not a transitive relationship)"
  1. Asking for a double verification
  • Final System prompt: "You are a logic bot designed to answer complex logic problems. You like to use a truth table to examine the alternatives. You only add the possible possibilities, being able to remove impossible situations. After your answer you double verify if your final sentence is valid"
  • Final User prompt: "User: Jack is looking at Anne. Anne is looking at George. Jack is married, George is not, and we don’t know if Anne is married. Is a married person looking at an unmarried person? (note that "looking at" is not a transitive relationship)"
  1. Reverting almost everything and appending asking to test all possibilities.
  • Final System prompt: "You are a logic bot designed to answer complex logic problems. You thoroughly test all possibilities to get to the right conclusions."
  • Final User prompt: "User: Jack is looking at Anne. Anne is looking at George. Jack is married, George is not, and we don’t know if Anne is married. Is a married person looking at an unmarried person? (note that "looking at" is not a transitive relationship)"
  1. Adjusting to change only the system prompt as it is a lesson about role playing, so, only the system prompt should be changed.
  • Final System prompt: "You are a logic bot designed to answer complex logic problems. You thoroughly test all possibilities to get to the right conclusions."
  • Final User prompt: Original prompt
  1. Appending what was in the user prompt in the system prompt
  • Final System prompt: "You are a logic bot designed to answer complex logic problems. You thoroughly test all valid logical possibilities on the unknown variables to get to the right conclusions. Remember "Looking at" is not a transitive relationship before arriving to conclusions."
  • Final User prompt: Original prompt
  1. This is actually a very simple problem, so changed the wording to "simple logic problems"
  • Final System prompt: "You are a logic bot designed to answer simple logic problems. You thoroughly test all valid logical possibilities on the unknown variables to get to the right conclusions. Remember "Looking at" is not a transitive relationship before arriving to conclusions."
  • Final User prompt: Original prompt
  1. Enforcing testing all possibilities and removing transitive relationship.
  • Final System prompt: "You are a logic bot designed to answer simple logic problems. You thoroughly test ALL valid logical possibilities on the unknown variables to get to the right conclusions."
  • Final User prompt: Original prompt
  1. Asking to evaluate the unknown variables. The first correct answer.
  • Final System prompt: "You are a logic bot designed to answer simple logic problems. You first evaluate which variables are unknown and thoroughly test ALL valid logical cases on the unknown variables to get to the right conclusions."
  • Final User prompt: Original prompt
  1. Removing " to get to the right conclusions" reverted to a wrong answer.
  • Final System prompt: "You are a logic bot designed to answer simple logic problems. You first evaluate which variables are unknown and thoroughly test ALL valid logical cases on the unknown variables."
  • Final User prompt: Original prompt
  1. Changing simple to complex again also reverted the result to a wrong answer. Then I gave up on getting a shorter solution and accepted 10 as the correct answer.
  • Final System prompt: "You are a logic bot designed to answer complex logic problems. You first evaluate which variables are unknown and thoroughly test ALL valid logical cases on the unknown variables to get to the right conclusions."
  • Final User prompt: Original prompt
Dominant language
Jupyter Notebook
Stars
39.5k
Forks
4.4k
PR merge metrics
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from anthropics/prompt-eng-interactive-tutorial

All issues in anthropics/prompt-eng-interactive-tutorial

Similar issues

More AI Infra & Agents issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.