NASDAQ-100 minute bars data preperation (inside the docker container) not working

[martinfrei-web]

Installing NASDAQ-100 minute bars doesn't seem to work as /data/equities/market/nasdaq100/minute_bars folder gets not created. Following all the output I get:

(ml4t) ubuntu@6f9caded0fc0:/app$ uv run python data/equities/market/algoseek_convert.py     --dataset nasdaq100-minute-bars --source data/nasdaq-100-constituents-taq-ext.zip
warning: `VIRTUAL_ENV=/Users/martin/dev/workspace/machine-learning-for-trading/.venv` does not match the project environment path `/opt/ml4t` and will be ignored; use `--active` to target the active environment instead
Using data path (ML4T_DATA_PATH): /data

============================================================
ALGOSEEK CONVERSION - nasdaq100-minute-bars
============================================================
[06:15:41] Source: data/nasdaq-100-constituents-taq-ext.zip
[06:15:41] Output: /data
[06:15:41] 505 days across 24 months -> /data/equities/market/nasdaq100/minute_bars

[martinfrei-web]

And I miss an empty entry for QUANDL_API_KEY= inside .env.example. Isn't that required anymore?

[stefan-jansen]

Thanks for the report - both points are real, and the first one is not what it looks like.

The conversion was running. Nothing is written until a whole month is concatenated, and that
write is what creates minute_bars/, so during the longest phase of the run there is no output and
no directory to check. Only a failed day logged anything; a successful one was silent. With 505
days across 24 months, the first month alone is ~21 days of ~504 symbol files, which is a long time
to sit at that last line with nothing to look at.

If the process was still alive, it was making progress. du -sh /data/equities/market/nasdaq100
after the first month lands is the quickest way to confirm that on the current release.

Fixed in 05c95d5: the converter now logs each day as it completes with its row count and its
position in the month, and announces the concatenate-and-sort step before it starts, so the quiet
phase is visible while it happens. The same commit drops each month's per-day frames once they are
concatenated - a month of minute bars for the full constituent list is the largest object the script
builds, so holding both copies at once was worth removing.

On QUANDL_API_KEY: you are right, and the template was the thing that was wrong. It is still
required - download.py tells you to add it and the loader declares it - so its absence from
.env.example was a straight omission. Added in the same commit, with a note that
NASDAQ_DATA_LINK_API_KEY is read as an alias.

One thing that is not a problem, since it appears in your output: the VIRTUAL_ENV warning means
uv ignored your host's .venv path and used the container's /opt/ml4t, which is what you want
inside Docker.

Both changes are on a branch and will reach main with the next merge.

[martinfrei-web]

Thanks for looking into this.

The conversion was running. Nothing is written until a whole month is concatenated, and that
write is what creates minute_bars/, so during the longest phase of the run there is no output and
no directory to check.

What I used to observe was that the process exited after the last line I pasted above, so the log I saw was actually

(ml4t) ubuntu@6f9caded0fc0:/app$ uv run python data/equities/market/algoseek_convert.py     --dataset nasdaq100-minute-bars --source data/nasdaq-100-constituents-taq-ext.zip
warning: `VIRTUAL_ENV=/Users/martin/dev/workspace/machine-learning-for-trading/.venv` does not match the project environment path `/opt/ml4t` and will be ignored; use `--active` to target the active environment instead
Using data path (ML4T_DATA_PATH): /data

============================================================
ALGOSEEK CONVERSION - nasdaq100-minute-bars
============================================================
[06:15:41] Source: data/nasdaq-100-constituents-taq-ext.zip
[06:15:41] Output: /data
[06:15:41] 505 days across 24 months -> /data/equities/market/nasdaq100/minute_bars
(ml4t) ubuntu@6f9caded0fc0:/app$

So the process used to exit without any error and the target folderr were not created. On the branch you provided the fix this is the output I see:

root@a3f9111878d0:/app# uv run python data/equities/market/algoseek_convert.py --dataset nasdaq100-minute-bars --source data/nasdaq-100-constituents-taq-ext.zip
Using data path (ML4T_DATA_PATH): /data

============================================================
ALGOSEEK CONVERSION - nasdaq100-minute-bars
============================================================
[18:48:59] Source: data/nasdaq-100-constituents-taq-ext.zip
[18:48:59] Output: /data
[18:48:59] 505 days across 24 months -> /data/equities/market/nasdaq100/minute_bars
[18:49:01]   20200102 (1/21): 95,480 rows
[18:49:04]   20200103 (2/21): 95,105 rows
[18:49:06]   20200106 (3/21): 95,550 rows
[18:49:08]   20200107 (4/21): 98,157 rows
[18:49:10]   20200108 (5/21): 95,228 rows
[18:49:12]   20200109 (6/21): 95,577 rows
[18:49:17]   20200110 (7/21): 94,358 rows
[18:49:21]   20200113 (8/21): 95,702 rows
[18:49:23]   20200114 (9/21): 95,591 rows
root@a3f9111878d0:/app#

So the process now again exits too early and nothing is being created, but this time we see it more explicilty.

One thing that is not a problem, since it appears in your output: the VIRTUAL_ENV warning means
uv ignored your host's .venv path and used the container's /opt/ml4t, which is what you want
inside Docker.

There I used devcontainer inside VSCode which is much more beneficial compared to Jupyter server approach, e.g. inside VSCode I can click on a class name of ml4t.XXX library and jump right into it's source, also debugging is much more easier, something not that practical using Jupyter server approach.

But this time I didn't do that as you see by the logs above and launched the container by hand, logged into it and ran the command above.

BTW I need to stick to Docker approach because I am on macOS Intel. I also quickly tested on macOS Silicon as well, building uvand running on the host itself, and I confirm there the same command works perfectly! But inside docker it doesn't seem to work, and I have enough free diskspace on my macOS Intel laptop.

Are you able to reproduce this behaviour?

[stefan-jansen]

You are right, and my previous answer was wrong on both counts.

The correction first. I said a month is ~21 days of ~504 symbol files. It is ~100 files: 505 is the archive's day count, and your own log settles it, since 95,480 rows/day is about 100 symbols over 960 extended-hours minutes. I used that inflated figure to argue the run was fine when it was not.

What is actually happening. convert_nasdaq100 accumulated every day of a month in a list and only concatenated and sorted it at the end, so the staged frames, the concatenation and the sort result were all alive together. That is roughly three times the month, about 3.5 GB for a full one. The exit with no traceback and no shell message is what a SIGKILL looks like from under uv run: uv is the intermediate parent, so bash never prints Killed.

The part that made it unrecoverable is that the resume check is per month. Your kill at day 9 threw away all nine days, and every retry restarted the same month and died in the same place. That is why it reads as "not working" rather than "slow", and why 05c95d5 did not help: it made the failure visible without changing what the run holds.

It also explains the asymmetry you found. Natively on Apple Silicon there is no cgroup limit and the month fits; in Docker the ceiling is Docker Desktop's VM allocation, not the Mac's RAM.

The fix is in #708. Each day is written out and dropped as it is parsed, and the month is assembled ten symbols at a time. Peak drops to one day while parsing and ~665 MB while assembling, the output is row-for-row identical in the same order as before, and an interrupted run now resumes at the day it stopped on instead of restarting the month. data/README.md gains the 2 GB floor and a note that the exit status is the first thing to check.

Three things that would confirm the cause rather than leave it inferred, if you have a moment:

1. echo $? immediately after the exit. 137 is the kill.
2. docker inspect <container> --format '{{.State.OOMKilled}}'.
3. Docker Desktop -> Settings -> Resources -> Memory, whatever it is set to on that Intel Mac.

If it turns out not to be memory, the fix in #708 is still worth having but I am looking in the wrong place, so the numbers matter.

[stefan-jansen]

The converter fix is merged: 852a465 on main (PR #708).

convert_nasdaq100 now writes each day to a staging parquet and assembles the month from those files ten symbols at a time, instead of holding every day in memory to concatenate and sort at the end. Two consequences for the failure you hit:

• Peak memory is now one day of bars while parsing plus one batch while assembling, roughly 1.3 GB for the NASDAQ-100 archive rather than about three copies of the month.
• A run that is killed partway keeps the days it already parsed. Re-running resumes at the day it stopped on. Previously the resume check was per month, so a kill discarded every parsed day and a retry died at the same place however many times you tried it.

data/README.md now states the 2 GB minimum, and that in Docker the ceiling is Docker Desktop's own memory allocation (Settings -> Resources -> Memory) rather than the host's RAM.

The echo $? / docker inspect --format "{{.State.OOMKilled}}" / Docker memory-setting answers are still worth having when you get a chance - they would confirm the kill was memory rather than something else - but the fix does not depend on them. Please pull main and re-run; if it still stops, the log now names the day and month it was on, which will say where to look.