Skip to content

Customer reqs/8596 speed up otanalytics by using vectorized operations - #741

Merged
briemla merged 137 commits into
mainfrom
customer-reqs/8596-speed-up-otanalytics-by-using-vectorized-operations
Oct 27, 2025
Merged

Customer reqs/8596 speed up otanalytics by using vectorized operations#741
briemla merged 137 commits into
mainfrom
customer-reqs/8596-speed-up-otanalytics-by-using-vectorized-operations

Conversation

@briemla

@briemla briemla commented Sep 18, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

OP#8596
OP#8685
OP#8740

Summary

This pull request introduces extensive improvements and changes to a Python-based analytics project. The changes include significant shift towards leveraging Polars (polars library), replacing older methods with Polars-based implementations for track datasets and geometries. Additionally, tighter type hinting, improved support for metadata handling, and enabling .feather file processing are included.

  • Introduced Polars support (polars) for track datasets & geometry operations.
  • Replaced older PandasTrackDataset with PolarsTrackDataset where applicable.
  • Introduced functionalities to convert .ottrk data into .feather file formats with metadata export.
  • Added utilities & tests comparing Polars vs Pandas implementations.
  • Cleaned up redundant methods (e.g., TrackToVideoRepository) and optimized repository logic.

Main Changes

Dataset Enhancements:

  • Added PolarsTrackDataset backed by PolarsTrackIdSet using polars.Series.
  • Enhanced geometry operations via PolarsTrackGeometryDataset.
  • Comprehensive replacement of PandasTrackDataset in applicable usage areas.

Metadata Handling:

  • .Feather parser (FeathersParser) support for tracks, with backward compatibility for .ottrk.
  • Detection metadata now managed efficiently across the system.

Bug Fixes & Tests:

  • Fixed logic around geometries, ensuring comprehensive intersection validations.
  • Added several benchmarks for new datasets and .feather-based parsing.

External Libraries:

  • Added dependencies polars and pyarrow to leverage Arrow/Feather formats effectively.

Critical Notes

  • General project performance likely impacted positively, especially for large datasets.
  • Introduced benchmarks explicitly comparing legacy methods against Polars-based workflows.

briemla added 30 commits May 14, 2025 16:12
@briemla
Add intersection of track segments with a single line
418e911
@briemla
Add test cases
831208d
@briemla
Introduce constants
854b7fd
@briemla
Use constants in tests
13915c0
@briemla
Separate functionality
016be45
@briemla
Intersect a polygon
d766152
@briemla
Use constant
aaf8f3e
@briemla
Move polygon
4c5e17d
@briemla
Implement intersection with first line segment of section
8ff8b27
@briemla
Process all legs of a segment
d19b5e1
@briemla
Fix type cast
86e9cc7
@briemla
Ensure offset is used
e7a2b29
@briemla
Add pure pandas and 2h test data
44e7e6c
@briemla
Add 4h benchmark
0b9b88d
@briemla
Extend benchmark to check pure pandas implementation
cc530d8
@briemla
Optimize calculation
15ee577
@briemla
Add offset
b594181
@briemla
Fix missing offset in intersection coordinate calculation
706357b
@briemla
Update todo comments
aa3bc49
@briemla
Fix different track id classes and add further todos
16f1255
@briemla
Use pandas geometry dataset
b52bc2e
@briemla
Remove shapely geometry store in base_application
9cba860
@briemla
Implement add_all, remove and get_for
a719ad5
@briemla
Fix broken test
476f2ad
@briemla
Clean up unused method
4bacd4f
@briemla
Add script to convert ottrk to feather format
f027d5d
@briemla
Directly use pandas version
078f2a4
@briemla
Implement feather track parser
18cc478
@briemla
Use feathers parser
fb0cf9e
@briemla
Convert a bunch of files to feathers
0793c73
briemla and others added 22 commits October 8, 2025 17:01
@briemla
Fix intersection calculation
526e5ff
@randy-seng
Refactor: Introduce constants for metadata keys and update references…
c4196dd 
… in `convert_ottrk_to_feathers.py` and `feathers_parser.py`.
@randy-seng
Refactor: Introduce constants for recorded start date in test metadat…
67804c6 
…a and add corresponding assertions in `test_feathers_parser.py`.
@randy-seng
Refactor: Streamline unpacking track IDs by using set comprehension f…
e3191c3 
…or `polars_track_id_set.py`.
@randy-seng
Remove redundant import EmptyTrackIdSet from polars_track_store.py.
e50b2fe
@randy-seng
Fix intersection calculation
00476f3
@randy-seng
Fix direction vector calculation in PolarsGeometryDataset
870d10d
@briemla
Add interpolated columns to output
cf67bba
@briemla
Fix wrong columns and extract hostname
bc36368
@briemla
Fix missing override when using filterd track dataset
86590eb
@briemla
Compare csv files using polars and differ between events and counts
fed50c6
@briemla
Fix track id for not cut tracks
72eed3f
@briemla
Merge branch 'main' into customer-reqs/8596-speed-up-otanalytics-by-u…
03f2d1f 
…sing-vectorized-operations
@briemla
Merge remote-tracking branch 'origin/customer-reqs/8596-speed-up-otan…
df148db 
…alytics-by-using-vectorized-operations' into customer-reqs/8596-speed-up-otanalytics-by-using-vectorized-operations
@briemla
OP#8596:
e7ebc4d 
Sort dataframe before processing
@briemla
Merge remote-tracking branch 'refs/remotes/origin/bug/8708-regression…
8cb25c0 
…-tests-do-not-verify-eventlist-export' into customer-reqs/8596-speed-up-otanalytics-by-using-vectorized-operations

# Conflicts:
#	OTAnalytics/plugin_prototypes/eventlist_exporter/eventlist_exporter.py
#	pyproject.toml
#	tests/regression_otanalytics.py
#	tests/utils/assertions.py
#	uv.lock
@randy-seng
Merge branch 'main' into customer-reqs/8596-speed-up-otanalytics-by-u…
6927543 
…sing-vectorized-operations
@briemla
OP#8596:
8100e71 
Compare events only as pandas data frames to be much faster.
@briemla
OP#8596:
1b26c44 
Sort polars track dataset the same way as with previous parsers
@briemla
OP#8596:
fc7f249 
Compare datetime values with 1ms precision.
@briemla
OP#8596:
3503baf 
Add debugging function
@briemla
Merge remote-tracking branch 'origin/customer-reqs/8596-speed-up-otan…
83d42b0 
…alytics-by-using-vectorized-operations' into customer-reqs/8596-speed-up-otanalytics-by-using-vectorized-operations
@briemla
briemla enabled auto-merge October 27, 2025 10:16
@randy-seng
randy-seng self-requested a review October 27, 2025 15:29
@briemla
briemla merged commit a387758 into main Oct 27, 2025
16 of 17 checks passed
@briemla
briemla deleted the customer-reqs/8596-speed-up-otanalytics-by-using-vectorized-operations branch October 27, 2025 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@randy-seng randy-seng randy-seng approved these changes

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@briemla @randy-seng