The improvement on the metric

[acse-yl2020]

Dear dspy experts,

Hope you are doing well.

When you are looking at this thread, my hands are probably on my head to scratch some ideas out of it or somewhere else to make myself calm down:)

I am writing this to kindly ask your opinions on the potential reasons for the optimization process not showing any improvement during iterations. The performance metric remains constant, i.e.,
Scores so far: [96.78, 96.78, 96.78, 96.78, 96.78, 96.78, 96.78, 96.78, 96.78, 96.78, 96.78, 96.78, 96.78]
Best score: 96.78
It looks super suspicious:)

My code is designed to tackle complex mathematical problems. I don’t expect it to output a simple “correct” or “incorrect” result. Instead, I aim to have it generate the reasoning steps involved in solving each problem. This approach leads to two problems:

1. The dataset I created does not contain definitive answers, only potential reasoning steps for solving the problems.
2. The evaluation metric is now based on reasoning assessments from the LLM I defined, rather than a binary correct/incorrect measure like the answer_exact_match function.
   Please bear with my code—I'm not sure where should my hands be put :)
   ------------------------------------------------------ signature------------------------------------------------------------

Define the signature

class ScoNeSignature(dspy.Signature):
question = dspy.InputField(desc="......")
answer = dspy.OutputField(desc="The detailed, mathematically rigorous solution to the optimization problem")
explanation = dspy.OutputField(desc="Step-by-step explanation of the solution process", default="")
equations = dspy.OutputField(desc="Relevant mathematical equations in LaTeX format", default="")

------------------------------------------------------ CoT----------------------------------------------------------------

Update the CoT class

class CoT(dspy.Module):
def init(self):
super().init()
self.generate_answer = dspy.ChainOfThought(ScoNeSignature)

def forward(self, question=None):
    prompt = f"""

    {question}

    Begin your answer now:"""
    
    result = self.generate_answer(question=prompt)
    return dspy.Prediction(
        answer=result.answer,
        explanation=result.explanation,
        equations=result.equations
    )

-------------------------------------------------------metric---------------------------------------------------------------

def metric(gold, pred, trace=None):
if hasattr(gold, 'questions'):
question = gold.questions
gold_answer = gold.answers
else:
raise AttributeError("Gold object doesn't have 'questions' attribute")

question = gold.questions
gold_answer = gold.answers
predicted_answer = pred.answer if hasattr(pred, 'answer') else pred.answers

prompts = {
    'relevance': f"----my prompt----.\nQuestion: {question}\nPredicted answer: {predicted_answer}",
    'mathematical_rigor': f"----my prompt----\nPredicted answer: {predicted_answer}",
    'completeness': f"E----my prompt----.\nQuestion: {question}\nPredicted answer: {predicted_answer}",
    'clarity': f"----my prompt----.\nPredicted answer: {predicted_answer}",
    'comparison_to_gold': f"----my prompt----.\nQuestion: {question}\nGold answer: {gold_answer}\nPredicted answer: {predicted_answer}"
}

scores = {}
with dspy.context(lm=gpt4T):
    for aspect, prompt in prompts.items():
        try:
            result = dspy.Predict(ScoNeSignature)(question=prompt, context=None).answer
            scores[aspect] = extract_score(result)
        except Exception as e:
            print(f"Error processing {aspect}: {e}")

# Calculate weighted average score
weights = {
    'relevance': 0.3,
    'mathematical_rigor': 0.4,
    'completeness': 0.2,
    'clarity': 0.05,
    'comparison_to_gold': 0.05
}
final_score = sum(weights[aspect] * scores[aspect] for aspect in weights)
'''
if trace is not None:
    print(f"Question: {question}")
    print(f"Gold Answer: {gold_answer}")
    print(f"Predicted Answer: {predicted_answer}")
    for aspect, score in scores.items():
        print(f"{aspect.capitalize()} Score: {score:.2f}")
    print(f"Final Score: {final_score:.2f}")
'''
return final_score if trace is None else final_score >= 0.65

---------------------------------------------------# Set up the optimizer-------------------------------------------------------

config = dict(max_bootstrapped_demos=8, max_labeled_demos=8, num_candidate_programs=10, num_threads=4)
teleprompter = BootstrapFewShotWithRandomSearch(metric=metric, **config)

Compile and optimize the program

optimized_program = teleprompter.compile(CoT(), trainset=train_dataset)

What I'm currently considering is that the evaluation metric could be improved, as the LLM tends to give good scores to answers as long as the answer appears 'reasonable'. However, it seems that the modifications on the metric, i.e., promotor structure, do not provide any fix to the current situation. Do you have any suggestions? Or any suggestions in general?

Thanks 3000:)