Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions docs/systems/lmdb.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# LMDB Format

The format `lmdb` stores the frames of one or more systems in a single [LMDB](http://www.lmdb.tech/doc/) database, and can be loaded or dumped through {class}`dpdata.System`, {class}`dpdata.LabeledSystem`, and {class}`dpdata.MultiSystems`. The on-disk layout coincides with the LMDB datasets read by the DeePMD-kit data loader. Core fields use the plural on-disk names `coords`, `cells`, `energies`, `forces`, and `virials`, which the DeePMD-kit reader maps to its internal names. Registered additional fields use their `deepmd_name`.
The format `deepmd/lmdb` (with the backward-compatible alias `lmdb`) stores the frames of one or more systems in a single [LMDB](http://www.lmdb.tech/doc/) database, and can be loaded or dumped through {class}`dpdata.System`, {class}`dpdata.LabeledSystem`, and {class}`dpdata.MultiSystems`. The on-disk layout coincides with the LMDB datasets read by the DeePMD-kit data loader. Core fields use the plural on-disk names `coords`, `cells`, `energies`, `forces`, and `virials`, which the DeePMD-kit reader maps to its internal names. Registered additional fields use their `deepmd_name`.

In contrast to the directory-based `deepmd/npy` format, every frame is stored as an independent record indexed by a global frame number. Frames within one database may therefore differ in the number of atoms and in chemical composition, which is suited to data sets in which the number of frames per system is small.

Expand Down Expand Up @@ -29,17 +29,17 @@ A single system or a collection of systems is written to one database.
```python
import dpdata

dpdata.LabeledSystem("OUTCAR", fmt="vasp/outcar").to("lmdb", "data.lmdb")
dpdata.LabeledSystem("OUTCAR", fmt="vasp/outcar").to("deepmd/lmdb", "data.lmdb")

dpdata.MultiSystems(*systems).to("lmdb", "data.lmdb")
dpdata.MultiSystems(*systems).to("deepmd/lmdb", "data.lmdb")
```

The element table recorded in `type_map` defaults to the union of the elements present in the data. An explicit table may be supplied through the `type_map` argument, for example the full periodic table.

```python
from dpdata.periodic_table import ELEMENTS

dpdata.MultiSystems(*systems).to("lmdb", "data.lmdb", type_map=list(ELEMENTS))
dpdata.MultiSystems(*systems).to("deepmd/lmdb", "data.lmdb", type_map=list(ELEMENTS))
```

Frames are committed in batches of 1,000 by default. The `write_batch_size` argument changes the transaction size. If a transaction exceeds `map_size`, the map is enlarged and the same encoded batch is retried before frame counters are advanced. The destination must not exist unless `overwrite=True` is supplied.
Expand All @@ -50,13 +50,13 @@ Data are first written to a temporary sibling database. After closing it, dpdata

The `frame_system_ids` entry of the metadata records, for every frame, the index of the source system it belongs to. DeePMD-kit uses this partition for system-wise sampling, for example through `prob_sys_size`.

When a {class}`dpdata.MultiSystems` is dumped through `to("lmdb", ...)`, its frames are first grouped by chemical formula, and systems that share a formula are merged into a single entry. The resulting `frame_system_ids` therefore reflect the formula grouping rather than the original sources, and the number of systems may be smaller than the number of inputs.
When a {class}`dpdata.MultiSystems` is dumped through `to("deepmd/lmdb", ...)`, its frames are first grouped by chemical formula, and systems that share a formula are merged into a single entry. The resulting `frame_system_ids` therefore reflect the formula grouping rather than the original sources, and the number of systems may be smaller than the number of inputs.

When the original partition must be retained, the function {func}`dpdata.formats.lmdb.dump_systems` writes an ordered sequence of systems without formula merging; each input becomes one system, numbered in iteration order.
When the original partition must be retained, the function {func}`dpdata.formats.deepmd.lmdb.dump_systems` writes an ordered sequence of systems without formula merging; each input becomes one system, numbered in iteration order.

```python
import dpdata
from dpdata.formats.lmdb import dump_systems
from dpdata.formats.deepmd.lmdb import dump_systems

systems = [dpdata.LabeledSystem(d, fmt="deepmd/npy") for d in directories]
dump_systems(systems, "data.lmdb", type_map=["H", "C", "N", "O"])
Expand All @@ -80,21 +80,21 @@ Reading an LMDB through dpdata subsequently groups frames by composition and doe
```python
import dpdata

ms = dpdata.MultiSystems.from_file("data.lmdb", fmt="lmdb")
ms = dpdata.MultiSystems.from_file("data.lmdb", fmt="deepmd/lmdb")
```

Frames are grouped by composition, and each composition is returned as one system. Atom order is canonicalized by a stable sort on the global atom type; coordinates and all registered atomic fields are permuted consistently. Frames of one composition must have identical field sets and a consistent periodic-boundary condition. By default (`mixed_type=False`) the element table of each resulting system is restricted to the elements that the system contains. When `mixed_type=True`, every system retains the complete element set from the database. Loading through {class}`dpdata.MultiSystems` may normalize the order of `atom_names`, so callers should use element names rather than assume that stored numerical indices are retained.

The general {class}`dpdata.System` constructor applies its `type_map` argument after format parsing. Consequently, a direct single-system load with an explicit `type_map` retains that complete requested table even when `mixed_type=False`; this is standard `System` behavior. The compact/full distinction above describes the dictionaries yielded to {class}`dpdata.MultiSystems`.

```python
ms = dpdata.MultiSystems.from_file("data.lmdb", fmt="lmdb", mixed_type=True)
ms = dpdata.MultiSystems.from_file("data.lmdb", fmt="deepmd/lmdb", mixed_type=True)
```

A database that holds a single composition may also be read into a {class}`dpdata.LabeledSystem`. If the database contains several compositions, only the first can be represented by a single system and a warning is issued.

```python
ls = dpdata.LabeledSystem("data.lmdb", fmt="lmdb")
ls = dpdata.LabeledSystem("data.lmdb", fmt="deepmd/lmdb")
```

The reader loads all selected frames into memory. The default `max_frames=100000` guard rejects larger data sets before decoding; set it to `None` only when sufficient memory is available. Large training data sets should normally be consumed directly by the DeePMD-kit data loader.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -1099,21 +1099,21 @@ class LMDBFormat(Format):

>>> import dpdata
>>> ls = dpdata.LabeledSystem("OUTCAR", fmt="vasp/outcar")
>>> ls.to("lmdb", "data.lmdb")
>>> ls.to("deepmd/lmdb", "data.lmdb")

Write many systems into one LMDB, forcing a global type map::

>>> ms = dpdata.MultiSystems(s1, s2, s3)
>>> ms.to("lmdb", "data.lmdb", type_map=["H", "C", "N", "O"])
>>> ms.to("deepmd/lmdb", "data.lmdb", type_map=["H", "C", "N", "O"])

Read back as standard (per-composition) systems::

>>> ms = dpdata.MultiSystems.from_file("data.lmdb", fmt="lmdb")
>>> ms = dpdata.MultiSystems.from_file("data.lmdb", fmt="deepmd/lmdb")

Read back keeping the full global type map on every system::

>>> ms = dpdata.MultiSystems.from_file(
... "data.lmdb", fmt="lmdb", mixed_type=True
... "data.lmdb", fmt="deepmd/lmdb", mixed_type=True
... )

Note that loading through :class:`dpdata.MultiSystems` normalises the
Expand Down Expand Up @@ -1198,7 +1198,7 @@ def dump_systems(
"""Write an ordered sequence of systems, one ``frame_system_id`` each.

Unlike :meth:`to_multi_systems` (the path used by
``MultiSystems.to('lmdb', ...)``), the systems are **not** merged by
``MultiSystems.to('deepmd/lmdb', ...)``), the systems are **not** merged by
formula: every element of ``systems`` becomes exactly one source
system in the database, numbered ``0, 1, 2, ...`` in iteration order.
This preserves the system partition recorded in ``frame_system_ids``,
Expand Down Expand Up @@ -1452,7 +1452,7 @@ def _first_system(self, file_name, *, require_labeled: bool, **kwargs):
warnings.warn(
f"LMDB '{file_name}' contains more than one composition; only the "
"first is loaded into a single System. Use "
"dpdata.MultiSystems.from_file(..., fmt='lmdb') to load all of them.",
"dpdata.MultiSystems.from_file(..., fmt='deepmd/lmdb') to load all of them.",
stacklevel=2,
)
return first
Expand Down Expand Up @@ -2027,7 +2027,7 @@ def dump_systems(
Each element of ``systems`` is stored as a distinct source system,
numbered ``0, 1, 2, ...`` in iteration order, and recorded in the
``frame_system_ids`` metadata. In contrast to
``MultiSystems.to('lmdb', ...)``, systems are not merged by formula, so
``MultiSystems.to('deepmd/lmdb', ...)``, systems are not merged by formula, so
the system partition used by DeePMD-kit's ``prob_sys_size`` is kept.

Parameters
Expand Down Expand Up @@ -2057,7 +2057,7 @@ def dump_systems(
Examples
--------
>>> import dpdata
>>> from dpdata.formats.lmdb import dump_systems
>>> from dpdata.formats.deepmd.lmdb import dump_systems
>>> systems = [
... dpdata.LabeledSystem(d, fmt="deepmd/npy") for d in directories
... ]
Expand Down
4 changes: 3 additions & 1 deletion dpdata/plugins/lmdb.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
from __future__ import annotations

from dpdata.format import Format
from dpdata.formats.lmdb.format import LMDBFormat
from dpdata.formats.deepmd.lmdb.format import LMDBFormat

# Canonical name; ``lmdb`` is kept as a backward-compatible alias.
Format.register("deepmd/lmdb")(LMDBFormat)
Format.register("lmdb")(LMDBFormat)
Loading
Loading