Aren't these two supposed to give similar embedding?

[qubit0]

Thanks a lot for such a nice package.
I am just trying to understand MDE. Here's an experiment I am doing:

from sklearn.datasets import load_digits
digits = load_digits()

Method 1:

#Embedding using preserve neighbors
embedding = pymde.preserve_neighbors(digits.data).embed(verbose=False);
pymde.plot(embedding, color_by=digits.target,marker_size=8.0)

Method 2:

def get_edges(n_dissimilar_edges, sim_edge, weight, neg_weight=-1):
    similar_edges = sim_edge
    dissimilar_edges = pymde.preprocess.dissimilar_edges(
        digits.data.shape[0], num_edges=n_dissimilar_edges, similar_edges=similar_edges)

    edges = torch.cat([similar_edges, dissimilar_edges])
    weights = torch.cat([weight, neg_weight*torch.ones(dissimilar_edges.shape[0])])
    return edges, weights

knn_graph = pymde.preprocess.k_nearest_neighbors(digits.data, k=8, verbose=True)
The reason for k = 8 is:

no=digits.data.shape[0] # # of nodes
nc2=no * (no - 1) / 2
int(max(min(15, nc2 * 0.01 / no), 5))

n_similar = knn_graph.edges.shape[0]
n_dissimilar = n_similar

edges, weights = get_edges(n_dissimilar,knn_graph.edges,knn_graph.weights)
f = pymde.penalties.PushAndPull(
        weights=weights,
        attractive_penalty=pymde.penalties.Log1p,
        repulsive_penalty=pymde.penalties.Log,
    )
std_mde_knn = pymde.MDE(
        n_items=digits.data.shape[0],
        embedding_dim=2,
        edges=edges,
        distortion_function=f,
        constraint=pymde.Standardized(),
    )

However, I do not get a similar embedding.

[akshayka]

Thanks for trying out PyMDE.

What happens if in method 1 you add a Standardized constraint?

embedding = pymde.preserve_neighbors(digits.data, constraint=pymde.Standardized()).embed(verbose=False)

Or, alternatively, omit the constraint from method 2. If you do either of these things, I think the embeddings will be similar.

[qubit0]

I thought that method 1 by default used a Standardized constraint. That's why I have a Standardized constraint in method 2.

If I get rid of the Standardized constraint from method 2 then I get similar embedding.

• Is there a way to compare how similar these embeddings are? and is there a way to quantify how good an embedding is?

• Is there a reason for choosing a similar edge weight to be 1 and a dissimilar edge weight to be -1?

• pymde.preserve_neighbor by default puts -1 as edge weight for dissimilar edges and 1 as edge weight for similar edges, right?

[akshayka]

Method 1 only imposes a Standardized constraint if the repulsive penalty is None. By default a repulsive penalty is provided, and no Standardized constraint is imposed. Sorry if that was unclear.

Answers to your questions:

Is there a way to compare how similar these embeddings are?

• You can compare embeddings by aligning them and then either visually comparing or computing a norm of their difference: https://pymde.org/mde/index.html#comparing-embeddings.

is there a way to quantify how good an embedding is?

• It can be difficult to quantify exactly how good an embedding is, since ultimately what matters is if the embedding was useful for your application. However, there are some ways to at least sanity check embeddings: https://pymde.org/mde/index.html#sanity-checking-embeddings.

Is there a reason for choosing a similar edge weight to be 1 and a dissimilar edge weight to be -1?

• Choosing +1 and -1 weights is the simplest thing you might do. You could also try to do more intelligent things, but in practice the simple choices usually work well enough.

• See also this excerpt from our book, chapter 4 section 3:

pymde.preserve_neighbor by default puts -1 as edge weight for dissimilar edges and 1 as edge weight for similar edges, right?

• preserve_neighbors does something only slightly more intelligent than this: -1 if items u and v are dissimilar; +2 if u is a neighbor of v and v is a neighbor of u; and +1 if u is a neighbor of v but v is not a neighbor of u (or vice versa).

Hope that helps!

[qubit0]

Thanks a lot, @akshayka. Makes a lot of sense now.

One more question. I am trying to embed citeseer data and while doing so I get:
Your dataset appears to contain duplicated items (rows); when embedding, you should typically have unique items.

Also, the embedding is bad.

from torch_geometric.datasets import Planetoid
dataset = 'citeseer'
dataset1 = Planetoid("./", dataset)
data = dataset1[0]

embedding = pymde.preserve_neighbors(data.x).embed(verbose=False);
pymde.plot(embedding, color_by=data.y,marker_size=8.0,axis_limits=(-2,2))

Also, UMAP and t-SNE are quite different:

import numpy as np
import umap

reducer = umap.UMAP()
embedded = reducer.fit_transform(data.x.numpy())
pymde.plot(embedded, color_by=data.y,marker_size=8.0)

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2)
embedded_tsne = tsne.fit_transform(data.x.numpy())
pymde.plot(embedded_tsne, color_by=data.y,marker_size=8.0)