Using NTXentLoss with PairMarginMiner in different distance functions

[HaominJiang]

Hi, Kevin. @KevinMusgrave

I noticed that the default distance used by NTXentLoss is cosine similarity while that used by PairMarginMiner is L2.

Does it make sense that directly using NTXentloss and PairMarginMiner without modifying their default distance at the same time? Or I should change any one distance function to keep their distance computation consistent?

[KevinMusgrave]

Conceptually, it's nice if they use the same distance function. But in practice, I don't know if it matters.

[HaominJiang]

Thanks for @KevinMusgrave 's prompt reply.

Although in fact this combination of the loss and the miner with their different default distance formulas can work at the program level, still hoping that someone can notice this issue and theoretically explain whether such a method is reasonable.

There are 2 main concern about this topic:

• Since SimClr and MoCo both use cosine similarity for NTXentLoss, if I change it to L2, will it make the model performance worse (my model has the similar structure with SimClr and MoCo)?
• If L2 distance will make the performance worse, can I use the cosine similarity for PairMarginMiner? Although the default distance used by MultiSimilarityMiner is cosine similarity, which is the same with the NTXentLoss, PairMarginMiner is more suitable for controlling hard negative pairs in my task.

[KevinMusgrave]

Moving this to Discussions

[HaominJiang]

Hi, Kevin. @KevinMusgrave

I think this discussion can be closed. After my deeper research on NTXentLoss and SupConLoss, I found that both of them have "intrinsic ability to perform hard positive/negative mining" (chapter 3.2.2 from paper https://arxiv.org/abs/2004.11362). So maybe we don't need apply a miner along with these loss functions.

For a more detailed explanation, we could find that:

• When temperature τ gets larger, the function graph of q*k/ τ becomes flatter, which means the different negative samples will have a similar product to an anchor. In other words, the the loss function will ignore more difference between negative samples.
• When temperature τ gets smaller, the function graph of q* k/τ becomes sharper, which means the loss function should focus on more hard pairs to make the exp(q*k/τ) larger, then the loss can be closer to 0.