Skip to content

fix(bedrock): make llama2 BOS a string, not a 1-tuple - #2108

Open
Osamaali313 wants to merge 1 commit into
FoundationAgents:mainfrom
Osamaali313:fix/llama2-bos-token-string
Open

fix(bedrock): make llama2 BOS a string, not a 1-tuple#2108
Osamaali313 wants to merge 1 commit into
FoundationAgents:mainfrom
Osamaali313:fix/llama2-bos-token-string

Conversation

@Osamaali313

Copy link
Copy Markdown

Problem

messages_to_prompt_llama2 (metagpt/provider/bedrock/utils.py) defines its beginning-of-sequence token as a one-element tuple, not a string:

def messages_to_prompt_llama2(messages: list[dict]) -> str:
    BOS = ("<s>",)          # trailing comma -> tuple, not str
    ...
    prompt = f"{BOS}"       # renders the tuple repr

Because BOS is a tuple, prompt = f"{BOS}" interpolates its repr, so every Llama-2 / Mistral prompt begins with the literal text ('<s>',) instead of the <s> BOS token.

The sibling messages_to_prompt_llama3 in the same file does the same thing correctly with a plain string, which confirms the intent:

def messages_to_prompt_llama3(messages: list[dict]) -> str:
    BOS = "<|begin_of_text|>"   # string
    prompt = f"{BOS}"

The malformed prompt flows through BaseBedrockProvider.get_request_body for the Mistral and Meta (Llama-2) providers in bedrock_provider.py, so every such Amazon Bedrock request was sent a prompt starting with ('<s>',).

Reproduction

BOS = ("<s>",)
prompt = f"{BOS}" + "[INST] hi [/INST]"
prompt starts with
before (BOS = ("<s>",)) ('<s>',)[INST] hi [/INST]startswith("<s>")False
after (BOS = "<s>") <s>[INST] hi [/INST]startswith("<s>")True

Fix

Drop the trailing comma so BOS is the string it is used as:

BOS = "<s>"

One character. python -m py_compile passes.

messages_to_prompt_llama2 defined `BOS = ("<s>",)` — the trailing comma makes
it a one-element tuple, so `prompt = f"{BOS}"` renders the tuple repr and every
prompt starts with the literal text `('<s>',)` instead of the `<s>`
beginning-of-sequence token. The sibling messages_to_prompt_llama3 defines
`BOS = "<|begin_of_text|>"` as a plain string and interpolates it the same way,
confirming BOS is meant to be a string. The result feeds
BaseBedrockProvider.get_request_body for Mistral/Llama-2, so every such Bedrock
request was sent a malformed prompt. Drop the trailing comma.
Copilot AI review requested due to automatic review settings July 16, 2026 20:32

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants