Skip to content

Commit 856fab3

Browse files
authored
Include SSH logs in workspace diagnostics (#372)
Currently, workspace log collection includes only the generated support bundle and omits local CLI SSH logs needed to investigate connection failures. This commit adds an SSH log collector that reads the directory resolved by the CLI manager and copies matching session and rotated logs into workspace diagnostics. The collector applies a three-day cutoff and retains the last 50 MiB per file to limit archive size. The remote environment supplies its known SSH process ID so the collector includes the active log regardless of age without repeating process discovery. The support bundle collector now continues SSH log collection after bundle failures, and the diagnostic collectors include the matching session ID in warnings when available. The README documents the included files and fixed limits, while regression tests cover directory resolution, log selection, truncation, cancellation, and failure handling. - resolves https://linear.app/codercom/issue/JET-34
1 parent 64ed6e0 commit 856fab3

12 files changed

Lines changed: 803 additions & 23 deletions

‎CHANGELOG.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,19 @@
44

55
### Added
66

7-
- include a Coder support bundle for the selected workspace and agent only when using the environment's
8-
_Collect logs_ action in Toolbox
7+
- include a Coder support bundle for the selected workspace and agent only when using the environment's _Collect logs_
8+
action in Toolbox
9+
- include active and recent local CLI SSH logs in workspace log collection, using the configured or default CLI log
10+
directory
911

1012
### Changed
1113

1214
- store CLI session tokens in the OS keyring by default on supported platforms; set `useKeyring` to `false` to opt out
13-
- share the same session token between REST and CLI authentication, passing it through the environment and sanitizing logs
15+
- share the same session token between REST and CLI authentication, passing it through the environment and sanitizing
16+
logs
1417
- remove stored credentials on explicit logout, and remove old plaintext session files after successful keyring login
15-
- bound credential and header commands to 60 seconds while preserving unbounded workspace starts and complete progress lines
18+
- bound credential and header commands to 60 seconds while preserving unbounded workspace starts and complete progress
19+
lines
1620

1721
- upgraded the Toolbox plugin API, dropping support for Toolbox versions older than 3.7.2
1822
- display live workspace start output from the Coder CLI and poll build status and provisioner output while updating,

‎README.md‎

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -409,6 +409,33 @@ The plugin runs `coder support bundle` using the deployment's existing CLI login
409409
and its agent, when available. It places `coder-support.zip` inside the environment's diagnostic directory in the
410410
Toolbox log archive. Collection does not start a stopped workspace.
411411

412+
The same diagnostic directory also includes recent local CLI SSH logs under `coder-ssh/`, even if support bundle
413+
generation fails. The plugin uses the configured SSH log directory, then `CODER_SSH_LOG_DIR`, or otherwise the CLI's
414+
default state directory: `$XDG_STATE_HOME/coder/logs` when set to an absolute path, falling back to
415+
`~/.local/state/coder/logs` on Linux, `~/Library/Application Support/coder/logs` on macOS, or
416+
`%LOCALAPPDATA%/coder/logs` on Windows.
417+
418+
Only regular files directly inside the SSH log directory with these names are eligible:
419+
420+
- `coder-ssh*.log`: CLI SSH session logs.
421+
- `coder-ssh*.log.<number>`: rotated CLI SSH logs, where `<number>` contains only digits, such as `.log.1`.
422+
- `<PID>.log`: legacy SSH logs with a numeric filename, such as `12345.log`.
423+
424+
Collection skips subdirectories, symbolic links, and other filenames, including rotated legacy files such as
425+
`12345.log.1`. These local logs can include sessions for other workspaces that share the log directory.
426+
427+
The following fixed limits apply and are not configurable:
428+
429+
- **Age:** include files modified within the last **3 days (72 hours)**, based on the file's last-modified time.
430+
The identified active SSH session's log for the selected workspace/agent is included regardless of age.
431+
Rotated logs remain subject to the age cutoff. If the active log cannot be identified, the cutoff applies to
432+
every file.
433+
- **Size:** copy only the last **50 MiB (52,428,800 bytes) per file**, including the active log. This is a limit on
434+
each file's uncompressed contents, not on the total archive size.
435+
436+
These limits apply only to the collected copies; the original log files are unchanged. Missing or unreadable SSH
437+
logs do not prevent collecting the remaining diagnostics.
438+
412439
The bundle can contain deployment health and configuration, network diagnostics, workspace build logs, template
413440
source, and agent diagnostics. Available information depends on your permissions and workspace connectivity.
414441
Review the bundle before sharing, following
@@ -417,7 +444,7 @@ Review the bundle before sharing, following
417444
Support bundles require a Coder CLI that supports `coder support bundle` (Coder 2.10 or newer). Collection runs until
418445
the CLI finishes or you cancel log collection. If the CLI is unavailable or unsupported, or authentication or
419446
connectivity fails, the plugin removes any partial bundle and attempts to include
420-
`coder-support-error.txt` instead. Toolbox's other diagnostics remain available.
447+
`coder-support-error.txt` instead. Local SSH logs and Toolbox's other diagnostics remain available.
421448

422449
### HTTP Request Logging
423450

‎gradle.properties‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
version=0.9.4
1+
version=0.9.5
22
group=com.coder.toolbox
33
name=coder-toolbox

‎src/main/kotlin/com/coder/toolbox/CoderRemoteEnvironment.kt‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ import kotlinx.coroutines.Dispatchers
3838
import kotlinx.coroutines.Job
3939
import kotlinx.coroutines.channels.Channel
4040
import kotlinx.coroutines.delay
41+
import kotlinx.coroutines.ensureActive
4142
import kotlinx.coroutines.flow.MutableStateFlow
4243
import kotlinx.coroutines.flow.StateFlow
4344
import kotlinx.coroutines.flow.update
@@ -114,6 +115,8 @@ class CoderRemoteEnvironment(
114115
private val networkMetricsMarshaller = Moshi.Builder().build().adapter(NetworkMetrics::class.java)
115116
private val proxyCommandHandle = SshCommandProcessHandle(context)
116117
private var pollJob: Job? = null
118+
@Volatile
119+
private var sshPid: Long? = null
117120

118121
init {
119122
if (context.settingsStore.shouldAutoConnect(id)) {
@@ -128,8 +131,13 @@ class CoderRemoteEnvironment(
128131
internal fun currentSessionId(): SessionId? =
129132
agent?.let { SessionIdRegistry.findSession(workspace.name, it.name) }
130133

131-
override val diagnosticInfoCollector: DiagnosticInfoCollector =
132-
CoderSupportBundleCollector(context.logger) { outputFile ->
134+
override val diagnosticInfoCollector: DiagnosticInfoCollector
135+
get() = CoderSupportBundleCollector(
136+
context.logger,
137+
cli.getSshLogDirectory(),
138+
::currentSessionId,
139+
sshPid,
140+
) { outputFile ->
133141
cli.supportBundle(WorkspaceAddress.from(workspace, agent), outputFile)
134142
}
135143

@@ -312,6 +320,7 @@ class CoderRemoteEnvironment(
312320
// Toolbox can invoke this hook again while retrying without first reporting a disconnect.
313321
// Replace the previous poller so one environment never leaves multiple pollers behind.
314322
pollJob?.cancel()
323+
sshPid = null
315324
pollJob = pollNetworkMetrics()
316325
}
317326

@@ -323,6 +332,8 @@ class CoderRemoteEnvironment(
323332
val sessionId = currentSessionId()
324333
context.logger.debug(sessionId, "Searching SSH command's PID for workspace $id...")
325334
val pid = proxyCommandHandle.findByWorkspaceAndAgent(workspace, currentAgent)
335+
ensureActive()
336+
sshPid = pid
326337
if (pid == null) {
327338
context.logger.debug(sessionId, "No SSH command PID was found for workspace $id")
328339
delay(POLL_INTERVAL)
@@ -366,6 +377,7 @@ class CoderRemoteEnvironment(
366377
fun dispose() {
367378
pollJob?.cancel()
368379
pollJob = null
380+
sshPid = null
369381
isConnected.update { false }
370382
val removedSessionId = agent?.let {
371383
SessionIdRegistry.removeSession(workspace.name, it.name)
@@ -401,6 +413,7 @@ class CoderRemoteEnvironment(
401413
context.logger.info(sessionId, "Stopping the network metrics poll job for $id")
402414
pollJob?.cancel()
403415
pollJob = null
416+
sshPid = null
404417
connectionRequest.update { false }
405418
isConnected.update { false }
406419
if (isManual) {

‎src/main/kotlin/com/coder/toolbox/cli/CoderCLIManager.kt‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,12 @@ class CoderCLIManager(
129129
val localBinaryPath: Path = context.settingsStore.binPath(deploymentURL)
130130
val coderConfigPath: Path = context.settingsStore.dataDir(deploymentURL).resolve("config")
131131

132+
/** Returns the effective SSH log directory for this CLI's current settings. */
133+
internal fun getSshLogDirectory(): Path = getSshLogDirectory(
134+
context.settingsStore.sshLogDirectory,
135+
os = currentOs,
136+
)
137+
132138
private fun createDownloadService(): CoderDownloadService {
133139

134140
val okHttpClient = CoderHttpClientBuilder.default(context)
@@ -679,6 +685,45 @@ class CoderCLIManager(
679685
private const val CODER_SESSION_TOKEN_ENV_VAR = "CODER_SESSION_TOKEN"
680686

681687
internal fun supportsKeyringStorage(os: OS?): Boolean = os == OS.MAC || os == OS.WINDOWS
688+
689+
/** Resolves the log directory used by `coder ssh`, including its XDG state directory default. */
690+
internal fun getSshLogDirectory(
691+
configuredDirectory: String?,
692+
env: Map<String, String> = System.getenv(),
693+
os: OS? = getOS(),
694+
home: Path = Path.of(
695+
env[if (os == OS.WINDOWS) "USERPROFILE" else "HOME"].orEmpty().ifEmpty {
696+
System.getProperty("user.home")
697+
},
698+
),
699+
): Path {
700+
val directoryOverride = configuredDirectory?.takeIf { it.isNotBlank() }
701+
?: env["CODER_SSH_LOG_DIR"]?.takeIf { it.isNotEmpty() }
702+
if (directoryOverride != null) return Path.of(directoryOverride)
703+
704+
// The CLI uses github.com/adrg/xdg: XDG_STATE_HOME must be absolute after home expansion.
705+
val stateHome = absoluteStatePath(env["XDG_STATE_HOME"], home, os) ?: when (os) {
706+
OS.MAC -> home.resolve("Library/Application Support")
707+
OS.WINDOWS -> absoluteStatePath(env["LOCALAPPDATA"], home, os) ?: home.resolve("AppData/Local")
708+
else -> home.resolve(".local/state")
709+
}
710+
return stateHome.resolve("coder/logs")
711+
}
712+
713+
private fun absoluteStatePath(value: String?, home: Path, os: OS?): Path? {
714+
if (value.isNullOrEmpty()) return null
715+
val prefix = if (os == OS.WINDOWS) {
716+
"%USERPROFILE%".takeIf { value.startsWith(it) }
717+
} else {
718+
listOf("~", "\$HOME").firstOrNull { value.startsWith(it) }
719+
}
720+
val path = if (prefix == null) {
721+
Path.of(value)
722+
} else {
723+
home.resolve(value.removePrefix(prefix).trimStart('/', '\\'))
724+
}
725+
return path.takeIf { it.isAbsolute }
726+
}
682727
}
683728

684729
private fun globalConfigArgs(): List<String> = listOf("--global-config", coderConfigPath.toString())
Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
package com.coder.toolbox.diagnostics
2+
3+
import com.coder.toolbox.session.SessionId
4+
import kotlinx.coroutines.CancellationException
5+
import kotlinx.coroutines.CoroutineScope
6+
import kotlinx.coroutines.Dispatchers
7+
import kotlinx.coroutines.ensureActive
8+
import kotlinx.coroutines.withContext
9+
import java.nio.channels.FileChannel
10+
import java.nio.file.Files
11+
import java.nio.file.LinkOption.NOFOLLOW_LINKS
12+
import java.nio.file.Path
13+
import java.nio.file.StandardOpenOption.CREATE
14+
import java.nio.file.StandardOpenOption.READ
15+
import java.nio.file.StandardOpenOption.TRUNCATE_EXISTING
16+
import java.nio.file.StandardOpenOption.WRITE
17+
import java.nio.file.attribute.BasicFileAttributes
18+
import java.time.Instant
19+
20+
// Include current CLI session filenames and older PID-based logs.
21+
private val SSH_LOG_FILE = Regex("coder-ssh.*\\.log(?:\\.\\d+)?|\\d+\\.log")
22+
private const val SSH_LOG_MAX_AGE_SECONDS = 3L * 24 * 60 * 60
23+
private const val SSH_LOG_MAX_BYTES = 50L * 1024 * 1024
24+
25+
internal class CoderSshLogCollector(
26+
private val logger: CoderLogger,
27+
private val logDirectory: Path,
28+
) {
29+
suspend fun collect(
30+
logsRootFolder: Path,
31+
sessionId: SessionId? = null,
32+
activeSshPid: Long? = null,
33+
) = withContext(Dispatchers.IO) {
34+
// Older CLIs may not have created any session logs.
35+
if (Files.notExists(logDirectory)) return@withContext
36+
37+
val cutoff = Instant.now().minusSeconds(SSH_LOG_MAX_AGE_SECONDS)
38+
val entries = Files.newDirectoryStream(logDirectory).use { it.toList() }
39+
// CLI proxy log names end in the parent SSH PID; timestamps distinguish reused PIDs.
40+
val activeLog = activeSshPid?.let { pid ->
41+
entries.filter { entry ->
42+
val name = entry.fileName.toString()
43+
(name == "$pid.log" || (name.startsWith("coder-ssh") && name.endsWith("-$pid.log"))) &&
44+
Files.isRegularFile(entry, NOFOLLOW_LINKS)
45+
}.maxByOrNull { it.fileName.toString() }
46+
}
47+
for (entry in entries) {
48+
ensureActive()
49+
// An active session can remain quiet longer than the collection window.
50+
collectFile(entry, logsRootFolder.resolve("coder-ssh"), cutoff.takeUnless { entry == activeLog }, sessionId)
51+
}
52+
}
53+
54+
@Suppress("TooGenericExceptionCaught")
55+
private fun CoroutineScope.collectFile(source: Path, destination: Path, cutoff: Instant?, sessionId: SessionId?) {
56+
if (!SSH_LOG_FILE.matches(source.fileName.toString())) return
57+
var output: Path? = null
58+
try {
59+
val attributes = Files.readAttributes(source, BasicFileAttributes::class.java, NOFOLLOW_LINKS)
60+
val expired = cutoff != null && attributes.lastModifiedTime().toInstant().isBefore(cutoff)
61+
if (!attributes.isRegularFile || expired) return
62+
63+
FileChannel.open(source, READ, NOFOLLOW_LINKS).use { input ->
64+
val size = input.size()
65+
if (size > SSH_LOG_MAX_BYTES) {
66+
logger.warn(
67+
sessionId,
68+
"Truncating SSH log $source ($size bytes) to its last $SSH_LOG_MAX_BYTES bytes",
69+
)
70+
}
71+
Files.createDirectories(destination)
72+
val targetPath = destination.resolve(source.fileName)
73+
output = targetPath
74+
FileChannel.open(targetPath, CREATE, WRITE, TRUNCATE_EXISTING).use { target ->
75+
copyTail(input, target, size)
76+
}
77+
}
78+
} catch (ex: CancellationException) {
79+
output?.let { runCatching { Files.deleteIfExists(it) } }
80+
throw ex
81+
} catch (ex: Exception) {
82+
output?.let { runCatching { Files.deleteIfExists(it) } }
83+
logger.warn(sessionId, ex, "Could not collect SSH log $source")
84+
}
85+
}
86+
87+
private fun CoroutineScope.copyTail(input: FileChannel, target: FileChannel, size: Long) {
88+
var remaining = minOf(size, SSH_LOG_MAX_BYTES)
89+
var offset = size - remaining
90+
while (remaining > 0) {
91+
ensureActive()
92+
val copied = input.transferTo(offset, remaining, target)
93+
if (copied == 0L) break
94+
offset += copied
95+
remaining -= copied
96+
}
97+
}
98+
}
Lines changed: 25 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,58 @@
11
package com.coder.toolbox.diagnostics
22

3+
import com.coder.toolbox.session.SessionId
34
import com.jetbrains.toolbox.api.remoteDev.deploy.DiagnosticInfoCollector
45
import kotlinx.coroutines.CancellationException
56
import kotlinx.coroutines.Dispatchers
67
import kotlinx.coroutines.withContext
78
import java.nio.file.Files
89
import java.nio.file.Path
910

10-
/** Collects a Coder support bundle for a workspace and its optional agent. */
11+
/** Collects a workspace support bundle and recent local Coder SSH logs. */
1112
internal class CoderSupportBundleCollector(
1213
private val logger: CoderLogger,
14+
sshLogDirectory: Path,
15+
private val currentSessionId: () -> SessionId? = { null },
16+
private val sshPid: Long? = null,
1317
private val collectBundle: suspend (Path) -> Unit,
1418
) : DiagnosticInfoCollector {
19+
private val sshLogs = CoderSshLogCollector(logger, sshLogDirectory)
20+
1521
// Toolbox discards the additional-diagnostics directory if the collector throws.
1622
@Suppress("TooGenericExceptionCaught")
1723
override suspend fun collectAdditionalDiagnostics(logsRootFolder: Path) = withContext(Dispatchers.IO) {
18-
val bundle = logsRootFolder.resolve("coder-support.zip")
24+
val sessionId = currentSessionId()
1925
try {
2026
Files.createDirectories(logsRootFolder)
27+
collectSupportBundle(logsRootFolder, sessionId)
28+
sshLogs.collect(logsRootFolder, sessionId, sshPid)
29+
} catch (ex: CancellationException) {
30+
throw ex
31+
} catch (ex: Exception) {
32+
logger.warn(sessionId, ex, "Could not collect Coder workspace diagnostics")
33+
}
34+
}
35+
36+
// A failed bundle is replaced by an error report so the SSH logs are still collected.
37+
@Suppress("TooGenericExceptionCaught")
38+
private suspend fun collectSupportBundle(logsRootFolder: Path, sessionId: SessionId?) {
39+
val bundle = logsRootFolder.resolve("coder-support.zip")
40+
try {
2141
collectBundle(bundle)
2242
} catch (ex: CancellationException) {
2343
runCatching { Files.deleteIfExists(bundle) }
2444
throw ex
2545
} catch (ex: Exception) {
26-
logger.warn(ex, "Could not collect the Coder support bundle")
46+
runCatching { Files.deleteIfExists(bundle) }
47+
logger.warn(sessionId, ex, "Could not collect the Coder support bundle")
2748
runCatching {
28-
Files.deleteIfExists(bundle)
2949
Files.writeString(
3050
logsRootFolder.resolve("coder-support-error.txt"),
3151
"Coder support bundle collection failed (${ex.javaClass.simpleName}). " +
3252
"Check deployment connectivity, login, and CLI support for 'coder support bundle' " +
3353
"(Coder 2.10 or newer). Other Toolbox logs are still available.\n",
3454
)
35-
}.onFailure { logger.warn(it, "Could not write the Coder diagnostic failure report") }
55+
}.onFailure { logger.warn(sessionId, it, "Could not write the Coder diagnostic failure report") }
3656
}
37-
Unit
3857
}
3958
}

0 commit comments

Comments
 (0)