Skip to content

Fix wget host-folder resolution to prevent empty/zero-file website archives - #93

Merged
AhmadIbrahiim merged 3 commits into
masterfrom
copilot/fix-website-downloader-issue
May 15, 2026
Merged

Fix wget host-folder resolution to prevent empty/zero-file website archives#93
AhmadIbrahiim merged 3 commits into
masterfrom
copilot/fix-website-downloader-issue

Conversation

Copilot AI commented May 15, 2026

Copy link
Copy Markdown
Contributor

The downloader could report compression success while producing unusable output (Total Downloaded file: 0) because the archive source folder was derived unreliably from wget stderr. This change hardens folder resolution so archiving targets the actual downloaded host directory.

  • Downloader flow: normalize and parse wget stderr safely

    • Convert stderr chunks to string before parsing/emitting.
    • Replace brittle substring logic with explicit Resolving ... ( host extraction.
  • Archive source selection: add deterministic fallback

    • If host extraction from stderr is unavailable, derive folder name from the requested URL (URL.hostname[:port]).
    • Skip conversion/archive step when no valid folder name can be resolved, instead of proceeding with invalid input.
  • Behavioral outcome

    • Compression completion now maps to a real downloaded folder, avoiding false-success states that previously surfaced as zero downloaded files.
const resolvingMatch = responseText.match(/Resolving\s+([^\s]+)\s+\(/);
if (resolvingMatch && resolvingMatch[1]) {
  website = resolvingMatch[1];
}

const websiteFolder = website || getWebsiteFolderName(data.website);
if (!websiteFolder) {
  io.emit(data.token, { progress: "Unable to determine downloaded website folder." });
  return;
}
archiver(websiteFolder, io, data);

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • example.com
    • Triggering command: /usr/bin/wget wget -mkEpnp --no-if-modified-since REDACTED (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI linked an issue May 15, 2026 that may be closed by this pull request
Copilot AI and others added 2 commits May 15, 2026 17:11
Copilot AI changed the title [WIP] Fix website downloader not working anymore Fix wget host-folder resolution to prevent empty/zero-file website archives May 15, 2026
Copilot AI requested a review from AhmadIbrahiim May 15, 2026 17:14
@AhmadIbrahiim
AhmadIbrahiim marked this pull request as ready for review May 15, 2026 17:27
@AhmadIbrahiim
AhmadIbrahiim merged commit be3e857 into master May 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Not working anymore

2 participants