Supporting references

[szunami]

Hi! I recently picked up cached and it enabled me to very quickly set up some caches for hot functions. I think this project is awesome.

As I began to optimize the rest of my codebase, it quickly became clear that cloning cached values was a bottleneck (mostly just the memory allocation, I was storing Vec<String>). After some thought, I realized that my use case could live off of references to the cached values.

I believed that cached doesn't support this sort of behavior; is it something you would consider adding? I am happy to help with implementation / flesh out the proposal a bit if it seems in scope for this project.

Thanks for building this in the first place!

[jaemk]

This is tricky since the caches generated by the macros will always need to be at the global / static level. I'll need to sit on this for a little bit while I think about it - a possible solution could be to have a version/option of the macro which requires the cached-function to always be passed an explicit cache instance to use, something looking like:

#[cached(use_argument_cache="my_cache")]
fn calculate(my_cache: &mut SizedCache<&str, usize>, input: &str) -> usize {
    input.len()
}
let mut cache = SizedCache::with_size(200);
let res = calculate(&mut cache, "big");

But this feels like something that should be done with just a helper function that copies the macro's inner logic

fn calculate(input: &str) -> usize {
    input.len()
}

let mut cache = SizedCache::with_size(200);
let res = cached::with_cache(&mut cache, calculate, "input_str");
// except I guess it's gotta be a macro to support variadic args
let res = cached::with_cache!(&mut cache, calculate, "input_str");

Each of these would also require relaxing the constraints of the K type to not require Clone

[szunami]

Gotcha, yeah I figured there were good reasons for the current constraints / approach.

Can you clarify what challenges arise because the cache is static? I don't mean to disagree, I just don't have much experience in working with static variables in rust.

[jaemk]

The macros basically expand to

lazy_static! {
    static CACHE: Arc<Mutex<HashMap<K, V>>> = ...;
}

fn func(input: K) -> V {
    if let Some(v) = CACHE.lock().unwrap().get(&input) {
        return v.clone()
    }
    ... run logic & set value
}

So since the cache will outlive the function inputs and outputs, they need to be things that the cache can "own" and hand out .clone()s of. The Clone requirement is necessary for the cached macros, but it's not strictly necessary for the actual cache stores. Two of the store types do list Clone as a requirement, but I believe we can remove them (partly thanks to #59 ) so you can use the stores directly (without the macros) more freely.

[Pzixel]

You can use Arc<RwLock<HashMap<K, Arc<V>>>> to represent references. It's not ideal but it will work. Also RwLock is probably a better choice than a mutex

[NobodyXu]

@jaemk Maybe T: ‘static can be used, which requires T to be either owned or contains only reference to’static data.

[seanlinsley]

I'm currently working around this by leaking a static reference using Box::leak:

#[once(sync_writes = true)]
fn example() -> &'static Vec<i32> {
    Box::leak(Box::new(vec![]))
}

It'd be great if the Box calls were moved into the cached crate to simplify user code.

[jaemk]

Hi @seanlinsley, you can return Arc<T> rather than leaking a &'static: #[once] requires the value type to be Clone, which Arc<T> satisfies for any T, and a hit clones the pointer rather than the Vec:

#[once]
fn example() -> Arc<Vec<i32>> {
    Arc::new(vec![])
}

The same applies to #[cached]. In both cases a hit hands back the same allocation, not a copy of it.

On references generally, since that is what this issue asks for: the stores already hand out &V where they can. CachedPeek::cache_peek(&self, &Q) -> Option<&V> is implemented by all seven single-owner in-memory stores. What cannot return a reference:

• the macros, at all: the generated wrapper holds the cache lock in a local guard and returns your declared type, so a &V borrowed from that guard cannot escape it
• the sharded stores, for the same reason, once per shard
• the redis and redb stores, where the value is deserialized on each read, so there is no stored &V to borrow

That leaves get, which mutates (lru recency, refresh-on-hit ttl, hit/miss counters) and so takes &mut self. Making it take &self is tracked and unresolved: the counters are easy, but recency behind a RefCell trades a compile error for a runtime borrow panic, which is not a good trade in a library.