Use assistant role in examples for chat models

[barnybug]

Related to this ticket: #977 and the problem of chat models repeating parts of the prompt instead of returning the answer alone. One way to encourage chat models to adhere to the format is to format the chat history with few shot examples interleaving the assistant role for the answer itself (example below).

Taking the original poster's example as a single shot - GPT 3.5 consistently returns the answer alone as desired if we follow this pattern.

Though I'm not sure this is possible currently with DSPy?

from openai import OpenAI
client = OpenAI()

for i in range(10):
    resp = client.chat.completions.create(
      model="gpt-3.5-turbo",
      messages=[
        {"role": "user", "content": """Assess the quality of a tweet along the specified dimension.
    
    ---
    
    Follow the following format.
    
    Assessed Text: ${assessed_text}
    Assessment Question: ${assessment_question}
    Assessment Answer: Yes or No

    ---

    Assessed Text: "Language models have transformed the way we interact with technology, enabling machines to understand and generate human language like never before. From chatbots to translation services, these models are paving the way for a more intelligent and connected world. #LanguageModels #NLP #AI"
    Assessment Question: Does the assessed tweet make for a self-contained, engaging tweet?
    Assessment Answer:
    """}, {"role": "assistant", "content": "Yes"}, {"role": "user", "content": """
    ---
    
    Assessed Text: "Structured generation is the key to creating organized and impactful content. By carefully planning and designing templates, we can ensure that our message is clear and effective. #contentcreation #structuredgeneration"
    Assessment Question: Does the assessed tweet make for a self-contained, engaging tweet?
    Assessment Answer:"""}
      ]
    )
    print(f"Response: {repr(resp.choices[0].message.content)}")

Response: 'Yes'
Response: 'Yes'
Response: 'Yes'
Response: 'Yes'
Response: 'Yes'
Response: 'Yes'
Response: 'Yes'
Response: 'Yes'
Response: 'Yes'
Response: 'Yes'

Compared to simply embedding the example in the user part only 32/50 of responses followed the format desired, putting the answer in as assistant message, 50/50 follow the format.

[tom-doerr]

It's being worked on here: #1083

[barnybug]

Excellent, you read my mind! :-)
Happy to help test the branch.