Fused triton quantize-dequantize in forward helpers

[ElizaWszola]

Benchmark output:

==================================================================================================================================
SUMMARY
==================================================================================================================================
Config               Size            Fused (ms)   T+T (ms)     T+Py (ms)    vs T+T     vs T+Py    Correct   
----------------------------------------------------------------------------------------------------------------------------------
int8_tensor          4096x4096            0.33 ms      0.56 ms      0.55 ms   1.69x     1.68x   Yes       
int8_tensor          4096x11008           0.35 ms      0.61 ms      0.81 ms   1.72x     2.29x   Yes       
int8_tensor          8192x8192            0.38 ms      0.72 ms      0.79 ms   1.87x     2.06x   Yes       
int4_tensor          4096x4096            0.14 ms      0.27 ms      0.27 ms   1.87x     1.87x   Yes       
int4_tensor          4096x11008           0.35 ms      0.61 ms      0.65 ms   1.75x     1.87x   Yes       
int4_tensor          8192x8192            0.39 ms      1.67 ms      0.79 ms   4.28x     2.04x   Yes       
int8_channel         4096x4096            0.15 ms      0.23 ms      0.23 ms   1.55x     1.55x   Yes       
int8_channel         4096x11008           0.37 ms      0.58 ms      0.61 ms   1.58x     1.68x   Yes       
int8_channel         8192x8192            0.41 ms      0.70 ms      0.76 ms   1.69x     1.84x   Yes       
int4_channel         4096x4096            0.15 ms      0.23 ms      0.23 ms   1.55x     1.56x   Yes       
int4_channel         4096x11008           0.36 ms      0.58 ms      0.62 ms   1.60x     1.69x   Yes       
int4_channel         8192x8192            0.41 ms      0.70 ms      0.76 ms   1.72x     1.87x   Yes       
int8_g128            4096x4096            0.21 ms      0.30 ms      0.26 ms   1.38x     1.21x   Yes       
int8_g128            4096x11008           0.42 ms      0.64 ms      1.05 ms   1.50x     2.49x   Yes       
int8_g128            8192x8192            0.64 ms      0.97 ms      1.64 ms   1.51x     2.54x   Yes       
int4_g128            4096x4096            0.39 ms      0.57 ms      0.51 ms   1.46x     1.32x   Yes       
int4_g128            4096x11008           0.41 ms      0.62 ms      0.62 ms   1.50x     1.50x   Yes       
int4_g128            8192x8192            0.64 ms      0.97 ms      0.86 ms   1.52x     1.35x   Yes       
fp4_g16              4096x4096            0.38 ms      0.56 ms      0.51 ms   1.48x     1.34x   Yes       
fp4_g16              4096x11008           0.48 ms      0.66 ms      0.64 ms   1.38x     1.33x   Yes       
fp4_g16              8192x8192            0.63 ms      0.96 ms      0.85 ms   1.53x     1.35x   Yes       
fp4_g32              4096x4096            0.21 ms      0.29 ms      0.26 ms   1.39x     1.22x   Yes       
fp4_g32              4096x11008           0.46 ms      0.64 ms      0.62 ms   1.39x     1.36x   Yes       
fp4_g32              8192x8192            0.63 ms      0.97 ms      0.85 ms   1.53x     1.35x   Yes       
int4_g128_asym       4096x4096            0.27 ms      0.41 ms      0.43 ms   1.52x     1.60x   Yes       
int4_g128_asym       4096x11008           0.45 ms      0.65 ms      0.76 ms   1.45x     1.69x   Yes       
int4_g128_asym       8192x8192            0.88 ms      1.46 ms      1.28 ms   1.66x     1.46x   Yes       
int8_b128x128        4096x4096            0.33 ms      0.53 ms      0.52 ms   1.59x     1.57x   Yes       
int8_b128x128        8192x8192            0.43 ms      0.69 ms      0.77 ms   1.63x     1.80x   Yes  

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 793069ae-4339-4582-aba9-f699bec8c789

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ElizaWszola.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[yiliu30]

Hi @ElizaWszola, nice work! Would it make sense to add torch.compile as a baseline as well? It can automatically generate the Triton kernels and apply the necessary operator fusion.

[mergify]

The quality checks have failed. Please run make style and make quality under
the root directory to adddress the lint failures. You will need to install the
dev optional install to get the required linting packages.

[mergify]

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews

	Protection	Waiting on
🔴	Require one maintainer review	👀 reviews

🔴 Require one maintainer review

Waiting for any of

• approved-reviews-by=HDCharles
• approved-reviews-by=brian-dellabetta
• approved-reviews-by=dsikka
• approved-reviews-by=kylesayrs

This rule is failing.

All PRs must have at least one approving review from a maintainer before merging.

• any of:
  • approved-reviews-by=HDCharles
  • approved-reviews-by=brian-dellabetta
  • approved-reviews-by=dsikka
  • approved-reviews-by=kylesayrs
• #changes-requested-reviews-by = 0