Repository navigation
fix(gaxios): enforce maxContentLength on headerless responses in Bun #9773
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
a2900c6
0cd43b0
eb5273f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -728,6 +728,8 @@ export class Gaxios implements FetchCompliance { | |
| proxy?: string; | ||
| cert?: string; | ||
| key?: string; | ||
| size?: number; | ||
| maxContentLength?: number; | ||
| tls?: {cert?: string; key?: string}; | ||
| }) | ||
| : undefined; | ||
|
|
@@ -807,6 +809,7 @@ export class Gaxios implements FetchCompliance { | |
| } | ||
| throw err; | ||
| } | ||
| const maxSize = rawInit?.size ?? rawInit?.maxContentLength ?? 0; | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Generated] Compute |
||
| if ( | ||
| res?.body && | ||
| typeof Readable.fromWeb === 'function' && | ||
|
|
@@ -820,13 +823,23 @@ export class Gaxios implements FetchCompliance { | |
| const origArrayBuffer = res.arrayBuffer.bind(res); | ||
| const origBlob = res.blob.bind(res); | ||
| const readBuffer = async () => { | ||
| const chunks: Buffer[] = []; | ||
| for await (const chunk of nodeStream!) { | ||
| chunks.push( | ||
| Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk as Uint8Array), | ||
| ); | ||
| nodeStream ||= Readable.fromWeb(rawBody); | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Generated] Lazily initialize
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. out of date
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seems like a genuine bug. I notice that nodeStream wasn't even assigned before!
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Re: your comment on why Now that those guards check
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is good to know. Thank you! |
||
| const chunks: Uint8Array[] = []; | ||
| let totalBytes = 0; | ||
| for await (const chunk of nodeStream) { | ||
| const bytes = chunk as Uint8Array; | ||
| totalBytes += bytes.byteLength; | ||
| if (maxSize > 0 && totalBytes > maxSize) { | ||
| throw Object.assign( | ||
| new Error( | ||
| `content size at ${String((input as {url?: unknown})?.url ?? input)} over limit: ${maxSize}`, | ||
| ), | ||
| {type: 'max-size'}, | ||
| ); | ||
| } | ||
| chunks.push(bytes); | ||
| } | ||
| return Buffer.concat(chunks); | ||
| return Buffer.concat(chunks, totalBytes); | ||
| }; | ||
| Object.defineProperty(res, 'body', { | ||
| get() { | ||
|
|
@@ -837,23 +850,23 @@ export class Gaxios implements FetchCompliance { | |
| enumerable: true, | ||
| }); | ||
| res.text = async () => { | ||
| if (!nodeStream) return origText(); | ||
| if (!nodeStream && maxSize <= 0) return origText(); | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Generated] Route |
||
| return (await readBuffer()).toString('utf8'); | ||
| }; | ||
| res.json = async () => { | ||
| if (!nodeStream) return origJson(); | ||
| if (!nodeStream && maxSize <= 0) return origJson(); | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Generated] Route |
||
| return JSON.parse(await res.text()); | ||
| }; | ||
| res.arrayBuffer = async () => { | ||
| if (!nodeStream) return origArrayBuffer(); | ||
| if (!nodeStream && maxSize <= 0) return origArrayBuffer(); | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Generated] Route |
||
| const buf = await readBuffer(); | ||
| return buf.buffer.slice( | ||
| buf.byteOffset, | ||
| buf.byteOffset + buf.byteLength, | ||
| ); | ||
| }; | ||
| res.blob = async () => { | ||
| if (!nodeStream) return origBlob(); | ||
| if (!nodeStream && maxSize <= 0) return origBlob(); | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Generated] Route |
||
| const buf = await readBuffer(); | ||
| const contentType = res.headers?.get?.('content-type') ?? ''; | ||
| return new Blob([buf], contentType ? {type: contentType} : undefined); | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -343,6 +343,28 @@ describe('🥁 configuration options', () => { | |
| scope.done(); | ||
| }); | ||
|
|
||
| it('should enforce maxContentLength on responses without a Content-Length header', async () => { | ||
| const body = {hello: 'world'}; | ||
| for (const responseType of [ | ||
| 'json', | ||
| 'text', | ||
| 'arraybuffer', | ||
| 'blob', | ||
| ] as const) { | ||
| // nock's .reply(200, body) omits Content-Length unless .replyContentLength() | ||
| // or an explicit 'content-length' header is specified, so this exercises | ||
| // the stream byte-counting limit ('over limit: 5') rather than the upfront | ||
| // Content-Length header check ("Response's `Content-Length` is over the limit."). | ||
| const scope = nock(url).get('/').reply(200, body); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. By default, To ensure the response is truly headerless and the stream limiter is exercised, explicitly set the const scope = nock(url)
.get('/')
.reply(200, body, {'transfer-encoding': 'chunked'});
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Note on the
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Generated] Thank you for pointing that out! To make it unambiguous in the test itself that the headerless stream byte-counting path is exercised rather than the upfront |
||
| await assert.rejects( | ||
| request({url, maxContentLength: 5, responseType}), | ||
| (err: Error) => | ||
| err instanceof GaxiosError && /over limit: 5/.test(err.message), | ||
| ); | ||
| scope.done(); | ||
| } | ||
| }); | ||
|
|
||
|
Comment on lines
+346
to
+367
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Generated] Add unit test verifying that |
||
| it('should support redirects by default', async () => { | ||
| const body = {hello: '🌎'}; | ||
| const scopes = [ | ||
|
|
@@ -658,6 +680,12 @@ describe('🥁 configuration options', () => { | |
| const jsonRes = await request({url}); | ||
| assert.deepStrictEqual(jsonRes.data, responseBody); | ||
|
|
||
| await assert.rejects( | ||
| request({url, maxContentLength: 2}), | ||
| (err: Error) => | ||
| err instanceof GaxiosError && /over limit: 2/.test(err.message), | ||
| ); | ||
|
|
||
|
Comment on lines
+683
to
+688
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Generated] Add unit assertions in the Bun fetch test verifying that |
||
| globalThis.fetch = (async () => { | ||
| throw {code: 'ECONNRESET', message: 'socket hang up'}; | ||
| }) as typeof fetch; | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[Generated] Declare
sizeandmaxContentLengthonrawInitin#createBunFetch()so the Bun fetch wrapper can read the configured byte limit.