Skip to content

Increase resources, k8s setting to handle overwhelming request - #7948

Open
0yukali0 wants to merge 1 commit into
flyteorg:mainfrom
0yukali0:flyte-7939
Open

Increase resources, k8s setting to handle overwhelming request#7948
0yukali0 wants to merge 1 commit into
flyteorg:mainfrom
0yukali0:flyte-7939

Conversation

@0yukali0

Copy link
Copy Markdown
Contributor

Tracking issue

Close #7939

Why are the changes needed?

flyte-benchmark triggers overloading problem of k8s API server, OOM of rustfs and flute-binary.

What changes were proposed in this pull request?

Set default reserved resources for flyte and rustfs.

How was this patch tested?

Running experiment with parameters with -k 10 -n 2000 to trigger the issue.
I recommended that environment 20core and 28G to run the benchmark with parameters -k 25 -n 2000 in devbox.

# git clone flyte-benchmark
uv run scripts/v2/swarm.py    --k 10 --n 2000 #  9core 26G
uv run scripts/v2/swarm.py    --k 25 --n 2000 #  24core 64G

Labels

  • fixed: For any bug fixed.

Setup process

My environments includes 9 core, 26G memory and 100G disk. (Colima instance)

  1. Setting resources in _common.py in benchmark.
def task_env(name):
    """Parent environment for one benchmark shape."""
    return flyte.TaskEnvironment(
        name=name,
        image=image,
        resources=flyte.Resources(cpu=("50m", "80m"), memory=("300Mi", "512Mi")),
        depends_on=[sleep_env],
    )
  1. Setting up devbox.
# git clone flyte
make devbox-build
make devbox-run

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Related #7757

Stack

If you do use git town to manage PR Stacks, the stack relevant to this PR
will show below. Otherwise, you can ignore this section.

Docs link

Signed-off-by: Yuteng Chen <a08h0283@gmail.com>
@0yukali0
0yukali0 marked this pull request as ready for review September 2, 2026 22:27
port: http
periodSeconds: 1
resources:
reqeusts:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
reqeusts:
requests:

@popojk popojk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to be be a typo in values.yaml, making the resources settings not valid in actual deployment. Could you please:

  1. Fix the typo and test it again
  2. Run make devbox-build to generate new manifest files under docker/devbox-bundled/manifests

Thanks!

@kumare3

kumare3 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Are you running the benchmark on devbox?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Fix] Increasing k8s setting to handle overwhelming requests in devbox.

3 participants