Worker pool?

[ronag]

Creating a new worker for every invocation can be rather slow when dispatching small tasks. How about adding a pool of workers?

[developit]

This could probably be adapted (and simplified) to suit:
https://gist.github.com/developit/65a2212731f6b00a8aaa55d70c594f5c

FWIW, it's best to only invoke greenlet once per function - generally at definition time. Dynamically creating workers to do single tasks will always have a nontrivial performance overhead.

[developit]

Just going to clarify for folks finding this issue - Greenlet does re-use the worker thread for all calls, but only if you hoist the greenlet() definition so it's being called one time for the given async function:

// BAD: this creates a new worker every time getPage() is called:
export function getPage(url) {
  const getJson = greenlet(async url => {
    return await (await fetch(url)).json();
  });
  return getJson(url);
}

// GOOD: this uses the same worker for each call to getPage():
const getJson = greenlet(async url => {
  return await (await fetch(url)).json();
});
export function getPage(url) {
  return getJson(url);
}

[imedadel]

hey @developit, I wanted to offload some functions to a worker and I was thinking if this 👇 solution is good enough? (the alternative is using those postMessage thingies by hand, which I hate):

let dispatch = greenlet(async (action) => {
  switch (action.type) {
    case 'fetchGH': {
      let url = `https://api.github.com/users/${action.username}`
      let res = await fetch(url)
      let profile = await res.json()
      return profile.name
    }
    case 'sayHi': {
      return "Hi " + action.name
    }
    default: {
      throw new Error(`Unknown action ${action.type}`)
    }
  }
})

[developit]

@imedadel that works, yup! There's also workerize, which is basically like Greenlet but with support for "exporting" multiple methods from the worker.

[imedadel]

@developit I love workerize(-loader)! But it didn't play well with Next.js, so I ended up using worker-plugin (which is also by you? :o) along with comlink.

So, I guess, if anyone is trying to bundle workers in Next.js, use that combo, it works perfectly. Otherwise, stick to greenlet and/or workerize.

(you should probably mark the last 3 comments as off-topic :) )

[developit]

Oh I didn't mean the loader - there is a non-loader version of workerize that uses the same internals as Greenlet, works at runtime.

[imedadel]

I needed to import the AWS SDK, that's why it wasn't an option :(

PS. If anyone needs to bundle web workers in Next.js, use Parcel (microbundle had a babel-related error) to build your workers to the /public folder. Then use comlink.

Otherwise use greenlet/workerize.