Which package is this bug report for?
@crawlee/core (observed through @crawlee/cheerio / CheerioCrawler)
Issue description
Request.pushErrorMessage(error) assumes that every Error has a string stack. When an error has stack === null or stack === undefined, it appends that value to request.errorMessages, although RequestOptions.errorMessages only accepts strings.
The invalid request can then fail while Crawlee reclaims, persists, marks handled, or reconstructs it:
ArgumentError: (array `RequestOptions.errorMessages`) Expected values to be of type `string` but received type `null`
We encountered this after an HTTP transport emitted a response-stream/TLS error with a valid message but no stack. The persistence failure prevented the request from settling normally and, in a long-lived shared crawler runtime, blocked later work that was waiting for that runtime to drain.
This appears to be the same underlying problem reported in #1351. That report was closed after maintainers requested a reproduction. The sample below provides a deterministic reproduction and shows where the invalid state originates.
The current v3.18.1 implementation still selects errorOrMessage.stack without verifying that it is a string:
https://github.com/apify/crawlee/blob/v3.18.1/packages/core/src/request.ts#L344-L373
Expected behavior: pushErrorMessage() should always append a string. For an Error without a stack, it could fall back to error.message, then String(error), or omit a non-string value.
Code sample
import { Request } from "@crawlee/cheerio";
const request = new Request({ url: "https://example.com" });
const error = new Error("response stream failed");
// Some transport/runtime errors can arrive without a stack.
Object.defineProperty(error, "stack", { value: null });
request.pushErrorMessage(error);
console.log(request.errorMessages); // [null]
// Reproduces what happens when the invalid request is reconstructed/persisted.
new Request({
url: request.url,
errorMessages: request.errorMessages,
});
Result:
ArgumentError: (array `RequestOptions.errorMessages`) Expected values to be of type `string` but received type `null`
Package version
Reproduced with @crawlee/core@3.17.0 through @crawlee/cheerio@3.17.0. The same unchecked stack assignment is present in the latest v3.18.1 source linked above.
Node.js version
Production runtime: Bun 1.4.0. The reproduction does not rely on Bun-specific APIs; it only constructs a Crawlee Request and an Error whose stack is null.
Operating system
macOS arm64 for the minimal reproduction; Linux arm64 in the affected worker runtime.
I have tested this on the next release
Not tested.
Other context
Our downstream workaround uses Crawlee's public errorHandler hook to normalize non-string entries in request.errorMessages before request persistence. We are not patching Crawlee locally.
Which package is this bug report for?
@crawlee/core(observed through@crawlee/cheerio/CheerioCrawler)Issue description
Request.pushErrorMessage(error)assumes that everyErrorhas a stringstack. When an error hasstack === nullorstack === undefined, it appends that value torequest.errorMessages, althoughRequestOptions.errorMessagesonly accepts strings.The invalid request can then fail while Crawlee reclaims, persists, marks handled, or reconstructs it:
We encountered this after an HTTP transport emitted a response-stream/TLS error with a valid message but no stack. The persistence failure prevented the request from settling normally and, in a long-lived shared crawler runtime, blocked later work that was waiting for that runtime to drain.
This appears to be the same underlying problem reported in #1351. That report was closed after maintainers requested a reproduction. The sample below provides a deterministic reproduction and shows where the invalid state originates.
The current v3.18.1 implementation still selects
errorOrMessage.stackwithout verifying that it is a string:https://github.com/apify/crawlee/blob/v3.18.1/packages/core/src/request.ts#L344-L373
Expected behavior:
pushErrorMessage()should always append a string. For anErrorwithout a stack, it could fall back toerror.message, thenString(error), or omit a non-string value.Code sample
Result:
Package version
Reproduced with
@crawlee/core@3.17.0through@crawlee/cheerio@3.17.0. The same unchecked stack assignment is present in the latestv3.18.1source linked above.Node.js version
Production runtime: Bun 1.4.0. The reproduction does not rely on Bun-specific APIs; it only constructs a Crawlee
Requestand anErrorwhose stack is null.Operating system
macOS arm64 for the minimal reproduction; Linux arm64 in the affected worker runtime.
I have tested this on the
nextreleaseNot tested.
Other context
Our downstream workaround uses Crawlee's public
errorHandlerhook to normalize non-string entries inrequest.errorMessagesbefore request persistence. We are not patching Crawlee locally.