file discovery/metadata API

Author: zekageri

CHANGELOG.md

@@ -21,6 +21,9 @@ The format follows Keep a Changelog and the project adheres to Semantic Versioni
   - `writeFileFromPath(path, sourceFsPath, opts)` to avoid manual source `File` management.
 - New `examples/LargeFileStreaming` sketch showing chunked large-binary streaming with streaming hash verification.
 - New `examples/AsyncLargeFileUpload` sketch showing background large-binary upload with progress polling and hash verification.
+- File discovery APIs for persisted uploads:
+  - `getFileInfo(path)` for per-path metadata lookup
+  - `listFiles(prefix, recursive)` for directory-style discovery under `/_files`
 
 ### Changed
 - `init()` now skips collections listed in `delayedCollectionSyncArray` during eager preload; deferred collections are loaded on first periodic autosync tick (or first `syncNow()` when `autosync=false`) and still load immediately on first explicit `collection(name)` access.
@@ -37,6 +40,11 @@ The format follows Keep a Changelog and the project adheres to Semantic Versioni
 - Async upload state retention is now bounded: terminal upload states are kept only for a recent window of upload IDs.
 - Updated `examples/FileStreaming` and file storage tests to cover callback/path convenience write flows.
 
+### Fixed
+- Collection cleanup on sync now reports filesystem removal errors instead of silently succeeding when a collection directory or document file cannot be deleted.
+- Recursive collection and `dropAll()` cleanup now removes directories consistently across Arduino-ESP32 and ESP-IDF builds.
+- Added hardware tests that verify collection directories and document files actually disappear from LittleFS after sync-driven cleanup.
+
 ### Removed
 - Legacy `onSync(std::function<void()>)` callback API (breaking change). Use `onSyncStatus(...)` instead.
 

README.md

@@ -115,6 +115,9 @@ db.writeFileStream(
     },
     fileOpts
 );
+
+auto fileInfo = db.getFileInfo("notes/readme.txt");
+auto fileTree = db.listFiles("firmware", true);
 ```
 
 ## Gotchas
@@ -125,6 +128,8 @@ db.writeFileStream(
 - `writeFileStream()` and `readFileStream()` hold the filesystem lock while processing the stream; use reasonable chunk sizes and avoid blocking stream sources/sinks.
 - `writeFileStreamAsync()` runs producer callbacks on a background task; callbacks must be short and thread-safe.
 - `getFileUploadState(uploadId)` retains terminal states for a bounded number of recent uploads; older upload IDs eventually return `NotFound`.
+- Uploaded files are not surfaced as collections or snapshots; use `getFileInfo()` / `listFiles()` to inspect persisted file storage under `/_files`.
+- `dropCollection()` only schedules on-disk removal; the collection directory and document files are deleted on the next autosync pass or `syncNow()`.
 - `/_files` is an internal reserved directory used for file storage and cannot be used as a collection name.
 - `getSnapshot()` and `restoreFromSnapshot()` currently cover document collections only; file storage under `/_files` is not included.
 - `usePSRAMBuffers` affects ESPJsonDB-owned byte buffers, decoded `DocView` `JsonDocument` pools on ArduinoJson v7, and long-lived internal DB containers (collection/schema/upload/diag maps and queues). Public return containers like `readFile()` still use the existing API types.
@@ -137,6 +142,7 @@ db.writeFileStream(
 - `void onSyncStatus(std::function<void(const DBSyncStatus&)>)` – observe cold preload and `syncNow()` progress with stage/source/current collection counters.
 - `onSync(std::function<void()>)` was removed; migrate to `onSyncStatus(...)`.
 - Collection management: `collection(name)`, `dropCollection(name)`, `dropAll()`, `getAllCollectionName()`.
+  - `dropCollection(name)` removes in-memory state immediately and deletes the corresponding filesystem directory on the next autosync pass or explicit `syncNow()`.
 - Document helpers:
   - Create: `create`, `createMany` (JSON array) plus direct `Collection::create*` variants.
   - Read: `findById`, `findOne`, `findMany` (predicate or JSON filter) returning `DocView` so you can read/write lazily.
@@ -153,6 +159,8 @@ db.writeFileStream(
   - `cancelFileUpload(uploadId)`, `getFileUploadState(uploadId)` for async job control (terminal states are retained for a bounded recent window).
   - `writeFile(path, data, size)` / `readFile(path)` for direct byte buffers.
   - `writeTextFile(path, text)` / `readTextFile(path)` for UTF-8 or plain text payloads.
+  - `getFileInfo(path)` returns a JSON object with `path`, `name`, `exists`, `isDirectory`, and `size`.
+  - `listFiles(prefix, recursive)` returns a JSON document with `prefix`, `recursive`, and an `entries` array of file/directory metadata objects.
   - `fileExists(path)`, `fileSize(path)`, `removeFile(path)` for file lifecycle utilities.
   - File paths are relative to `/<baseDir>/_files` and path traversal segments are rejected.
   - `ESPJsonDBFileOptions`: `overwrite` and `chunkSize` controls for stream writes.

src/esp_jsondb/collection/collection.cpp

@@ -918,7 +918,9 @@ DbStatus Collection::flushDirtyToFs(const std::string &baseDir, bool &didWork) {
 			{
 				FrLock fs(g_fsMutex);
 				if (_fs->exists(path.c_str())) {
-					_fs->remove(path.c_str());
+					if (!_fs->remove(path.c_str())) {
+						return recordStatus({DbStatusCode::IoError, "document delete failed"});
+					}
 				}
 			}
 		}

src/esp_jsondb/db.cpp

@@ -7,7 +7,7 @@
 
 namespace {
 constexpr uint32_t kTaskStopTimeoutMs = 200;
-static void removeTree(fs::FS &fsImpl, const std::string &path);
+static DbStatus removeTree(fs::FS &fsImpl, const std::string &path);
 using DirEntry = std::pair<std::string, bool>;
 using DirEntryVector = JsonDbVector<DirEntry>;
 } // namespace
@@ -888,7 +888,10 @@ DbStatus ESPJsonDB::runSyncPass() {
 	DbStatus finalStatus{DbStatusCode::Ok, ""};
 	if (dropAll) {
 		if (_fs) {
-			removeTree(*_fs, _baseDir);
+			auto st = removeTree(*_fs, _baseDir);
+			if (!st.ok()) {
+				return setLastError(st);
+			}
 		}
 		auto st = ensureFsReady();
 		if (!st.ok()) {
@@ -1045,42 +1048,52 @@ static void listDirEntries(
 	d.close();
 }
 
-static void removeTree(fs::FS &fsImpl, const std::string &path) {
+static DbStatus removeTree(fs::FS &fsImpl, const std::string &path) {
 	// Check if path is a directory
 	bool isDir = false;
 	{
 		FrLock fs(g_fsMutex);
 		if (!fsImpl.exists(path.c_str()))
-			return;
+			return {DbStatusCode::Ok, ""};
 		File f = fsImpl.open(path.c_str());
 		if (f) {
 			isDir = f.isDirectory();
 			f.close();
+		} else {
+			return {DbStatusCode::IoError, "open path failed during recursive remove"};
 		}
 	}
 	if (!isDir) {
 		FrLock fs(g_fsMutex);
-		fsImpl.remove(path.c_str());
-		return;
+		if (!fsImpl.remove(path.c_str())) {
+			return {DbStatusCode::IoError, "remove file failed during recursive remove"};
+		}
+		return {DbStatusCode::Ok, ""};
 	}
 	// List children first without holding lock during recursion
 	DirEntryVector entries{JsonDbAllocator<DirEntry>(false)};
 	listDirEntries(fsImpl, path, entries);
 	for (auto &e : entries) {
 		if (e.second) {
-			removeTree(fsImpl, e.first);
+			auto st = removeTree(fsImpl, e.first);
+			if (!st.ok()) {
+				return st;
+			}
 		} else {
 			FrLock fs(g_fsMutex);
-			fsImpl.remove(e.first.c_str());
+			if (!fsImpl.remove(e.first.c_str())) {
+				return {DbStatusCode::IoError, "remove child file failed during recursive remove"};
+			}
 		}
 	}
 	// Finally remove the directory itself
 	{
 		FrLock fs(g_fsMutex);
-#ifdef ARDUINO_ARCH_ESP32
-		fsImpl.rmdir(path.c_str());
-#endif
+		if (!fsImpl.rmdir(path.c_str())) {
+			return {DbStatusCode::IoError, "remove directory failed during recursive remove"};
+		}
 	}
+	return {DbStatusCode::Ok, ""};
 }
 } // namespace
 
@@ -1089,9 +1102,11 @@ DbStatus ESPJsonDB::removeCollectionDir(const std::string &name) {
 	if (!dir.empty() && dir.back() != '/')
 		dir += '/';
 	dir += name;
-	if (_fs)
-		removeTree(*_fs, dir);
-	return setLastError({DbStatusCode::Ok, ""});
+	if (!_fs) {
+		return setLastError({DbStatusCode::InvalidArgument, "filesystem handle is null"});
+	}
+	auto st = removeTree(*_fs, dir);
+	return setLastError(st);
 }
 
 void ESPJsonDB::emitEvent(DBEventType ev) {

src/esp_jsondb/db.h

@@ -184,6 +184,8 @@ class ESPJsonDB {
 	readFileStream(const std::string &relativePath, Stream &out, size_t chunkSize = 512);
 	DbResult<std::vector<uint8_t>> readFile(const std::string &relativePath);
 	DbResult<std::string> readTextFile(const std::string &relativePath);
+	DbResult<JsonDocument> getFileInfo(const std::string &relativePath);
+	DbResult<JsonDocument> listFiles(const std::string &relativePrefix = "", bool recursive = true);
 
 	DbStatus removeFile(const std::string &relativePath);
 	DbResult<bool> fileExists(const std::string &relativePath);

src/esp_jsondb/db_files.cpp

@@ -17,6 +17,108 @@ std::string parentDirOf(const std::string &path) {
 	return path.substr(0, pos);
 }
 
+std::string fileNameOf(const std::string &path) {
+	auto pos = path.find_last_of('/');
+	if (pos == std::string::npos)
+		return path;
+	return path.substr(pos + 1);
+}
+
+struct FileEntryInfo {
+	std::string path;
+	bool isDirectory = false;
+	size_t size = 0;
+};
+
+DbStatus statFileEntry(
+    fs::FS &filesystem,
+    const std::string &absolutePath,
+    const std::string &relativePath,
+    FileEntryInfo &out
+) {
+	FrLock fs(g_fsMutex);
+	if (!filesystem.exists(absolutePath.c_str())) {
+		return {DbStatusCode::NotFound, "file not found"};
+	}
+	File file = filesystem.open(absolutePath.c_str(), FILE_READ);
+	if (!file) {
+		return {DbStatusCode::IoError, "open file info failed"};
+	}
+	out.path = relativePath;
+	out.isDirectory = file.isDirectory();
+	out.size = out.isDirectory ? 0 : file.size();
+	file.close();
+	return {DbStatusCode::Ok, ""};
+}
+
+void appendFileInfoJson(JsonArray entries, const FileEntryInfo &info) {
+	JsonObject entry = entries.add<JsonObject>();
+	entry["path"] = info.path.c_str();
+	entry["name"] = fileNameOf(info.path).c_str();
+	entry["exists"] = true;
+	entry["isDirectory"] = info.isDirectory;
+	entry["size"] = info.size;
+}
+
+DbStatus collectDirectoryEntries(
+    fs::FS &filesystem,
+    const std::string &absoluteDir,
+    const std::string &relativeDir,
+    bool recursive,
+    std::vector<FileEntryInfo> &entries
+) {
+	std::vector<std::pair<std::string, std::string>> pendingDirs;
+	{
+		FrLock fs(g_fsMutex);
+		File dir = filesystem.open(absoluteDir.c_str(), FILE_READ);
+		if (!dir || !dir.isDirectory()) {
+			if (dir)
+				dir.close();
+			return {DbStatusCode::NotFound, "file not found"};
+		}
+		for (File child = dir.openNextFile(); child; child = dir.openNextFile()) {
+			const bool isDirectory = child.isDirectory();
+			String rawName = child.name();
+			child.close();
+			std::string segment = rawName.c_str();
+			auto slash = segment.find_last_of('/');
+			if (slash != std::string::npos)
+				segment = segment.substr(slash + 1);
+			if (segment.empty())
+				continue;
+
+			FileEntryInfo info;
+			info.path = relativeDir.empty() ? segment : joinPath(relativeDir, segment);
+			info.isDirectory = isDirectory;
+			info.size = 0;
+			if (!isDirectory) {
+				const std::string childPath = joinPath(absoluteDir, segment);
+				File childFile = filesystem.open(childPath.c_str(), FILE_READ);
+				if (!childFile) {
+					dir.close();
+					return {DbStatusCode::IoError, "open child file info failed"};
+				}
+				info.size = childFile.size();
+				childFile.close();
+			}
+			entries.push_back(info);
+
+			if (recursive && isDirectory) {
+				pendingDirs.emplace_back(joinPath(absoluteDir, segment), info.path);
+			}
+		}
+		dir.close();
+	}
+
+	for (const auto &pending : pendingDirs) {
+		auto st = collectDirectoryEntries(filesystem, pending.first, pending.second, true, entries);
+		if (!st.ok())
+			return st;
+	}
+
+	return {DbStatusCode::Ok, ""};
+}
+
 DbStatus writeFromPullCb(
     fs::FS &filesystem,
     const std::string &finalPath,
@@ -426,6 +528,92 @@ DbResult<std::string> ESPJsonDB::readTextFile(const std::string &relativePath) {
 	return res;
 }
 
+DbResult<JsonDocument> ESPJsonDB::getFileInfo(const std::string &relativePath) {
+	DbResult<JsonDocument> res{};
+	auto ready = ensureReady();
+	if (!ready.ok()) {
+		res.status = setLastError(ready);
+		return res;
+	}
+
+	std::string normalized;
+	auto nst = normalizeFilePath(relativePath, normalized);
+	if (!nst.ok()) {
+		res.status = setLastError(nst);
+		return res;
+	}
+
+	FileEntryInfo info;
+	const auto st = statFileEntry(*_fs, joinPath(fileRootDir(), normalized), normalized, info);
+	if (!st.ok()) {
+		res.status = setLastError(st);
+		return res;
+	}
+
+	res.value["path"] = info.path.c_str();
+	res.value["name"] = fileNameOf(info.path).c_str();
+	res.value["exists"] = true;
+	res.value["isDirectory"] = info.isDirectory;
+	res.value["size"] = info.size;
+	res.status = setLastError({DbStatusCode::Ok, ""});
+	return res;
+}
+
+DbResult<JsonDocument>
+ESPJsonDB::listFiles(const std::string &relativePrefix, bool recursive) {
+	DbResult<JsonDocument> res{};
+	auto ready = ensureReady();
+	if (!ready.ok()) {
+		res.status = setLastError(ready);
+		return res;
+	}
+
+	std::string normalizedPrefix;
+	if (!relativePrefix.empty()) {
+		auto nst = normalizeFilePath(relativePrefix, normalizedPrefix);
+		if (!nst.ok()) {
+			res.status = setLastError(nst);
+			return res;
+		}
+	}
+
+	const std::string rootPath = fileRootDir();
+	const std::string targetPath =
+	    normalizedPrefix.empty() ? rootPath : joinPath(rootPath, normalizedPrefix);
+
+	FileEntryInfo targetInfo;
+	auto st = statFileEntry(*_fs, targetPath, normalizedPrefix, targetInfo);
+	if (!st.ok()) {
+		res.status = setLastError(st);
+		return res;
+	}
+
+	std::vector<FileEntryInfo> entries;
+	if (targetInfo.isDirectory) {
+		st = collectDirectoryEntries(*_fs, targetPath, normalizedPrefix, recursive, entries);
+		if (!st.ok()) {
+			res.status = setLastError(st);
+			return res;
+		}
+	} else {
+		entries.push_back(targetInfo);
+	}
+
+	std::sort(entries.begin(), entries.end(), [](const FileEntryInfo &lhs, const FileEntryInfo &rhs) {
+		return lhs.path < rhs.path;
+	});
+
+	res.value["prefix"] = normalizedPrefix.c_str();
+	res.value["recursive"] = recursive;
+	JsonArray entriesJson = res.value["entries"].to<JsonArray>();
+	for (const auto &entry : entries) {
+		appendFileInfoJson(entriesJson, entry);
+	}
+
+	res.status = setLastError({DbStatusCode::Ok, ""});
+	return res;
+}
+
 DbStatus ESPJsonDB::removeFile(const std::string &relativePath) {
 	auto ready = ensureReady();
 	if (!ready.ok())