You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Python bindings for the fast **light-weight** integer compression library [FastPFor](https://github.com/lemire/FastPFor): A research library with integer compression schemes. FastPFor is broadly applicable to the compression of arrays of 32-bit integers where most integers are small. The library seeks to exploit SIMD instructions (SSE) whenever possible. This library can decode at least 4 billions of compressed integers per second on most desktop or laptop processors. That is, it can decompress data at a rate of 15 GB/s. This is significantly faster than generic codecs like gzip, LZO, Snappy or LZ4.
5
7
6
8
# Authors
@@ -9,31 +11,72 @@ Daniel Lemire, Leonid Boytsov, Owen Kaser, Maxime Caron, Louis Dionne, Michel Le
9
11
10
12
# Installation
11
13
12
-
Bindings can be installed locally:
14
+
Install the latest release from PyPI:
15
+
16
+
```
17
+
pip install pyfastpfor
18
+
```
19
+
20
+
### Using uv (recommended)
21
+
22
+
[uv](https://github.com/astral-sh/uv) is a fast Python package installer and resolver.
23
+
24
+
To add PyFastPFor as a dependency to your project:
25
+
26
+
```
27
+
uv add pyfastpfor
28
+
```
29
+
30
+
Or to install it directly into your current environment:
31
+
32
+
```
33
+
uv pip install pyfastpfor
34
+
```
35
+
36
+
### Building from source
37
+
38
+
From the repository root, run:
39
+
13
40
```
14
41
cd python_bindings
15
-
pip install -r requirements.txt
16
-
sudo setup.py build install
17
42
```
18
-
or via pip:
43
+
44
+
With uv:
45
+
19
46
```
20
-
pip install pyfastpfor
47
+
uv build
48
+
uv pip install dist/*.whl
21
49
```
22
-
Due to some compilation quirks this currently seem to work with GCC only. I will fix it in some not so distant future. You may also need to install Python dev-files. On Ubuntu, for Python 3 you can do it as follows:
50
+
51
+
Or with standard tools:
23
52
24
53
```
25
-
sudo apt-get install python3-dev
54
+
pip install build
55
+
python -m build
56
+
pip install dist/*.whl
26
57
```
27
58
59
+
Debug information is disabled by default. To include it in a local build, set
60
+
`PYFASTPFOR_DEBUG_INFO=1` when building:
61
+
62
+
```
63
+
PYFASTPFOR_DEBUG_INFO=1 python -m build
64
+
```
65
+
66
+
The bindings build with GCC, Clang, or MSVC, on both x86-64 (SSE/AVX) and ARM/aarch64 (NEON, including Apple Silicon). You may also need to install Python dev-files. On Ubuntu, for Python 3 you can do it as follows:
67
+
68
+
```
69
+
sudo apt-get install python3-dev
70
+
```
28
71
29
72
# Documentation
30
73
31
-
The library supports all the codecs implemented in the original [FastPFor](https://github.com/lemire/FastPFor) library by July 2023. To get a list of codecs, use the function ``getCodecList``.
74
+
The library supports all the codecs implemented in the original [FastPFor](https://github.com/lemire/FastPFor) library (v0.5.0). To get a list of codecs, use the function ``getCodecList``.
32
75
33
-
Typical light-weight compression does not take context into account and, consequently, works well only for small integers. When integers are large, data differencing is a common trick to make integers small. In particular, we often deal with sorted lists of integers, which can be represented by differences between neighboring numbers.
76
+
Typical light-weight compression does not take context into account and, consequently, works well only for small integers. When integers are large, data differencing is a common trick to make integers small. In particular, we often deal with sorted lists of integers, which can be represented by differences between neighboring numbers.
34
77
35
-
The smallest differences (**fine** deltas) are between adjacent numbers. Respective differencing and difference inverting functions are ``delta1'' and``prefixSum1''.
78
+
The smallest differences (**fine** deltas) are between adjacent numbers. Respective differencing and difference inverting functions are ``delta1'' and``prefixSum1''.
36
79
37
-
However, we can do reasonably well, we compute differences between numbers that are four positions apart (**coarse** deltas). Such differences can be computed and inverted more efficiently. Respective differencing and difference inverting functions are ``delta4'' and``prefixSum4''.
80
+
However, we can do reasonably well, we compute differences between numbers that are four positions apart (**coarse** deltas). Such differences can be computed and inverted more efficiently. Respective differencing and difference inverting functions are ``delta4'' and``prefixSum4''.
38
81
39
-
Examples of three common use scenarios (no differencing, coarse and fine deltas) are outlined in [this Python notebook](python_bindings/examples.ipynb).
82
+
Examples of three common use scenarios (no differencing, coarse and fine deltas) are outlined in [this Python notebook](python_bindings/examples.ipynb).
0 commit comments