Divide the validation set

[wzicheng100-coder]

Hi All
Could I ask if I used DP and trained a potential function model from scratch? After comparing it with DFT, the accuracy was acceptable. However, later I discovered that no validation set was set up. How should I remedy this situation? Or, is it necessary to use DP to train the potential function and set up a validation set?

[dosubot[bot]]

Hi @wzicheng100-coder! I'm Dosu and I'm helping the DeepModeling team.

A validation set is optional but recommended in DeePMD-kit. The validation_data parameter defaults to None, so training works fine without it [1]. However, it's useful for monitoring whether your model is overfitting during training.

Since your model already shows acceptable accuracy compared to DFT, the situation isn't critical — but here's how you can remedy it:

Option 1: Retrain with a validation set

Re-split your dataset (e.g., 80/20) and retrain with the validation_data block added to your input JSON:

{
  "training": {
    "training_data": {
      "systems": ["training_data/"],
      "batch_size": "auto"
    },
    "validation_data": {
      "systems": ["validation_data/"],
      "batch_size": "auto",
      "numb_btch": 1
    },
    "numb_steps": 100000
  }
}

[2]

Option 2: Restart from checkpoint with validation added

You can continue training from your existing checkpoint while adding a validation set using --restart:

dp train input.json --restart model.ckpt

This restores the model and optimizer state and continues training, now with validation monitoring enabled [3] [4].

Best practice for splitting data: Merge all your data and do a random split using dpdata [5]:

import dpdata, numpy as np

all_data = dpdata.LabeledSystem("your_data", fmt="deepmd/npy")
idx = np.arange(len(all_data))
np.random.shuffle(idx)
split = int(0.8 * len(all_data))

all_data.sub_system(idx[:split]).to("deepmd/npy", "training_data")
all_data.sub_system(idx[split:]).to("deepmd/npy", "validation_data")

Summary: If your model already agrees well with DFT, you're likely fine. The main value of a validation set is to detect overfitting during training. If you want to confirm your model isn't overfit, the simplest check is to evaluate it on structures that were not in your training set and compare with DFT.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[dosubot[bot]]

@wzicheng100-coder When validation_data is not specified in your input JSON, DeePMD-kit simply skips validation entirely and trains using only the training data [1]. Training loss is still calculated and reported in lcurve.out, but you won't see any validation loss columns — only training metrics.

Default behavior without validation_data:

The validation_data parameter defaults to None [2]. Your input JSON would look something like this (with no validation_data block):

{
  "training": {
    "training_data": {
      "systems": ["./data/system_1", "./data/system_2"],
      "batch_size": "auto"
    },
    "numb_steps": 100000
  }
}

In this case, all the systems listed under training_data are used for training — none are automatically held out for validation.

Important note: If you're familiar with DeePMD-kit v1, the old numb_test parameter used to control validation. In v2.0+, numb_test is deprecated and has no effect [1]. Validation must now be configured explicitly via the validation_data block.

So to directly answer your question: the model was trained by simply optimizing the loss function on all provided training data without any held-out validation check. The training still converges and produces a model — it just means you have no built-in way to detect overfitting during the training process. Since you verified accuracy against DFT independently, you effectively performed external validation, which is a reasonable substitute.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[wzicheng100-coder]

@dosu
Using dpgen to train the potential function model, compared with the advantages of traditional machine learning in training potential functions, and why there is no need to add a validation set and a test set

[dosubot[bot]]

@wzicheng100-coder Here's an explanation of how DP-GEN differs from traditional ML potential training and why the validation/test set requirement is relaxed:

Advantages of DP-GEN over traditional ML potential training:

1. Concurrent learning instead of sequential learning — Traditional ML potentials require you to first generate a large ab initio MD (AIMD) dataset, then split it into train/validation/test, and train. DP-GEN instead generates data on-the-fly through an iterative explore→select→label→train cycle, building an optimal dataset that is both representative and as small as possible [1].

2. Active exploration of configuration space — Rather than relying on expensive, lengthy AIMD trajectories for data, DP-GEN uses fast DP-based MD to explore new regions of configuration space that the model doesn't yet cover well [2].

3. Dramatically reduced DFT cost — Only uncertain structures are sent for DFT labeling, so you avoid redundant first-principles calculations on well-learned configurations [2].

Why DP-GEN reduces the need for a separate validation/test set:

DP-GEN trains an ensemble of models (typically 4) with different random seeds. The model deviation — the standard deviation of force predictions across this ensemble — acts as a built-in uncertainty measure [3] [4]:

• Low deviation → model is confident and accurate for that structure
• Intermediate deviation → structure is selected for DFT labeling (new training data)
• High deviation → likely non-physical, discarded

This model deviation mechanism effectively serves the same purpose as a validation set: it tells you where the model is unreliable and provides a natural convergence criterion (the process stops when no more high-deviation structures are found) [5].

However, an important nuance: Model deviation measures uncertainty, not absolute accuracy. It's still good practice to:

• Use validation_data within each DeePMD-kit training step to monitor overfitting during training
• Evaluate your final converged model against DFT on held-out structures before production use

In your case, since you already compared with DFT and found acceptable accuracy, you've effectively performed external validation — which confirms your model is reliable for the configurations you tested.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[dosubot[bot]]

@wzicheng100-coder This is a commonly reported issue. The problem is that DP-GEN transforms paths internally — it doesn't use your absolute paths directly in the training configuration. Instead, it creates symbolic links and converts paths to relative references within the training working directory [1] [2].

The validation_data paths you specify in param.json may not be handled through the same symlink mechanism that training_data uses, which causes the "file not found" error even though the files exist at those locations [3].

Possible solutions:

1. Check if training runs on remote nodes — If you're submitting to an HPC cluster, the compute nodes may not have access to those absolute paths. The data needs to be accessible from wherever the training job actually runs.

2. Use init_data_prefix + relative paths — Instead of absolute paths in validation_data, try structuring your validation data under a path that's relative to your init_data_prefix, so DP-GEN can properly symlink it.

3. Manually place validation data — Since DP-GEN uses all its generated data for training [3], validation data must be prepared externally. You may need to place it where the training directory can resolve the symlinked paths (i.e., ensure the data is accessible via ../data.init/ relative to the training task directory).

Could you share the exact error message? Also, are you running on an HPC cluster where the training jobs execute on different nodes? That would help narrow down whether this is a path resolution issue or an accessibility issue.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[dosubot[bot]]

@wzicheng100-coder The error confirms what I suspected — the training job is looking for the validation data at the absolute path /home/hanye/ljy_52/ljy/mg_h/run/vailtion_data/data.000 but cannot find it on the remote compute node.

Since you're submitting for remote computing, the issue is that the remote machine does not have access to that path. The absolute path exists on your local/login node, but when the training job runs on the compute node, it cannot reach that file system location.

To fix this, you have two options:

1. Ensure the path is accessible on the compute node — If your cluster uses a shared file system (e.g., NFS, Lustre), verify that /home/hanye/ljy_52/ljy/mg_h/run/vailtion_data/data.000 is mounted and accessible on the compute nodes. You can test this by SSH-ing into a compute node and checking if the path exists.

2. Include validation data in the files that get transferred — If DP-GEN transfers files to the compute node for the job, you need to make sure the validation data is included in that transfer. You may need to place your validation data within the DP-GEN working directory structure so it gets copied/linked to the remote node along with the training data.

Also, I notice a possible typo in your path: vailtion_data — did you mean validation_data? Please double-check that the directory name on disk matches exactly what's in your param.json.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[njzjz-bot]

One correction to the Dosu replies: model deviation does not replace either validation or independent testing.

Model deviation measures disagreement among models trained on the same data. The models can have the same systematic error and therefore agree closely while all being wrong. Thus, low model deviation is useful for active-learning selection, but it is not evidence of absolute DFT accuracy.

The practical remedy does not necessarily require discarding the existing model:

1. Prepare representative structures that were not used for training and, ideally, were not selected during the DP-GEN loop.
2. Label them consistently with DFT.
3. Test energy, force, and virial/stress errors separately by composition and thermodynamic region, and validate the physical properties needed for production.
4. Retrain with a validation split if you need training-time overfitting diagnostics; retrain/add data if the independent test exposes a coverage or accuracy problem.

If the earlier “comparison with DFT” used genuinely unseen structures, that was already an external test. If it used training structures, it was only a training-fit check.

Coding agent: Codex
Codex version: codex-cli 0.149.0
Model: gpt-5.6-sol
Reasoning effort: xhigh