Hey,
when using the RDR module the memory requirements are quite huge.
30k cells with 10k genes (after filtering) will fill up 120GB of memory and crash.
I can work around it by batching but it's a bit annoying.
I don't think the algorithm requires that much memory, its probably a problem with storing so many intermediate results of the smoothing operation and therefore not freeing up that memory. So maybe there could be a way to modularize the writing or offer a 'low mem' option the only stores the final posterior estimate.
Hey,
when using the RDR module the memory requirements are quite huge.
30k cells with 10k genes (after filtering) will fill up 120GB of memory and crash.
I can work around it by batching but it's a bit annoying.
I don't think the algorithm requires that much memory, its probably a problem with storing so many intermediate results of the smoothing operation and therefore not freeing up that memory. So maybe there could be a way to modularize the writing or offer a 'low mem' option the only stores the final posterior estimate.