Error on create/check, is my repository corrupted?

[BBaoVanC]

I'm using borgmatic, and I got a notification from BorgBase that I haven't had any backups for a day. I looked at my logs and I see that it's failing with an error. Here's what I get when trying with -v 2:

ae6e261dfd3d:/# borgmatic --stats --files -v 2
Ensuring legacy configuration is upgraded
/etc/borgmatic.d/config.yaml: No commands to run for pre-everything hook
borg --version --debug --show-rc
ssh://[REDACTED]: Running actions for repository
/etc/borgmatic.d/config.yaml: No commands to run for pre-actions hook
/etc/borgmatic.d/config.yaml: No commands to run for pre-backup hook
ssh://[REDACTED]: Creating archive
ssh://[REDACTED]: Calling postgresql_databases hook function remove_database_dumps
ssh://[REDACTED]: Removing PostgreSQL database dumps
ssh://[REDACTED]: Calling mysql_databases hook function remove_database_dumps
ssh://[REDACTED]: Removing MySQL database dumps
ssh://[REDACTED]: Calling mongodb_databases hook function remove_database_dumps
ssh://[REDACTED]: Removing MongoDB database dumps
ssh://[REDACTED]: Calling sqlite_databases hook function remove_database_dumps
ssh://[REDACTED]: Removing SQLite database dumps
ssh://[REDACTED]: Calling mysql_databases hook function dump_databases
ssh://[REDACTED]: Dumping MySQL databases
ssh://[REDACTED]: Querying for "all" MySQL databases to dump
mysql --host mysql --protocol tcp --user root --skip-column-names --batch --execute show schemas
ssh://[REDACTED]: Dumping MySQL database "all" to /root/.borgmatic/mysql_databases/mysql/all
mysqldump --add-drop-database --host mysql --protocol tcp --user root --databases creative_coreprotect luckperms --result-file /root/.borgmatic/mysql_databases/mysql/all
ssh://[REDACTED]: Collecting special file paths
borg create --exclude-from /tmp/tmpngsnyhen --compression auto,zstd --one-file-system --read-special --lock-wait 60 --list --filter AMEx- ssh://[REDACTED]::borgmatic-{now:%Y-%m-%dT%H:%M:%S.%f} /mnt/source /root/.borgmatic --dry-run --list
borg create --exclude-from /tmp/tmpngsnyhen --compression auto,zstd --one-file-system --read-special --lock-wait 60 --list --filter AMEx- ssh://[REDACTED]::borgmatic-{now:%Y-%m-%dT%H:%M:%S.%f} /mnt/source /root/.borgmatic --stats --debug --show-rc
using builtin fallback logging configuration
33 self tests completed in 0.11 seconds
SSH command line: ['ssh', '[REDACTED]@[REDACTED].repo.borgbase.com', 'borg', 'serve', '--debug']
Remote: using builtin fallback logging configuration
Remote: borg selftest disabled via BORG_SELFTEST env variable
Remote: using builtin fallback logging configuration
Remote: Initialized logging system for JSON-based protocol
Remote: Resolving repository path b'/./repo'
Remote: Resolved repository path to '/srv/repos/[REDACTED]/repo'
Remote: Verified integrity of /srv/repos/[REDACTED]/repo/index.126865
Remote: Traceback (most recent call last):
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1453, in get_fd
    ts, fd = self.fds[segment]
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/lrucache.py", line 21, in __getitem__
    value = self._cache[key]  # raise KeyError if not found
KeyError: 126862
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/remote.py", line 240, in serve
    res = f(**args)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1204, in get
    return self.io.read(segment, offset, id)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1569, in read
    fd = self.get_fd(segment)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1455, in get_fd
    fd = open_fd()
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1436, in open_fd
    fd = open(self.segment_filename(segment), 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/srv/repos/[REDACTED]/repo/data/126/126862'
RemoteRepository: 198 B bytes sent, 4.34 kB bytes received, 4 messages sent
Traceback (most recent call last):
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1453, in get_fd
    ts, fd = self.fds[segment]
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/lrucache.py", line 21, in __getitem__
    value = self._cache[key]  # raise KeyError if not found
KeyError: 126862
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/remote.py", line 240, in serve
    res = f(**args)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1204, in get
    return self.io.read(segment, offset, id)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1569, in read
    fd = self.get_fd(segment)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1455, in get_fd
    fd = open_fd()
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1436, in open_fd
    fd = open(self.segment_filename(segment), 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/srv/repos/[REDACTED]/repo/data/126/126862'
Borg server: Platform: Linux box-us7.borgbase.com 4.18.0-425.19.2.el8_7.x86_64 #1 SMP Fri Mar 17 01:52:38 EDT 2023 x86_64
Borg server: Linux: Unknown Linux
Borg server: Borg: 1.2.4  Python: CPython 3.9.13 msgpack: 1.0.3 fuse: None [pyfuse3,llfuse]
Borg server: PID: 1381721  CWD: /srv/repos/[REDACTED]
Borg server: sys.argv: ['/opt/venvs/borg-LATEST/bin/borg', 'serve', '--restrict-to-repository', '/srv/repos/[REDACTED]/repo', '--storage-quota', '100000M']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --debug'
Platform: Linux ae6e261dfd3d 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64
Linux: Unknown Linux
Borg: 1.2.4  Python: CPython 3.11.2 msgpack: 1.0.5 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 51  CWD: /
sys.argv: ['/usr/local/bin/borg', 'create', '--exclude-from', '/tmp/tmpngsnyhen', '--compression', 'auto,zstd', '--one-file-system', '--read-special', '--lock-wait', '60', '--list', '--filter', 'AMEx-', 'ssh://[REDACTED]::borgmatic-{now:%Y-%m-%dT%H:%M:%S.%f}', '/mnt/source', '/root/.borgmatic', '--stats', '--debug', '--show-rc']
SSH_ORIGINAL_COMMAND: None
terminating with error status, rc 2
ssh://[REDACTED]: Error running actions for repository
Command 'borg create --exclude-from /tmp/tmpngsnyhen --compression auto,zstd --one-file-system --read-special --lock-wait 60 --list --filter AMEx- ssh://[REDACTED]::borgmatic-{now:%Y-%m-%dT%H:%M:%S.%f} /mnt/source /root/.borgmatic --stats --debug --show-rc' returned non-zero exit status 2.
/etc/borgmatic.d/config.yaml: No commands to run for on-error hook
/etc/borgmatic.d/config.yaml: An error occurred
/etc/borgmatic.d/config.yaml: No commands to run for post-everything hook

summary:
/etc/borgmatic.d/config.yaml: An error occurred
ssh://[REDACTED]: Error running actions for repository
...
Traceback (most recent call last):
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/remote.py", line 240, in serve
    res = f(**args)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1204, in get
    return self.io.read(segment, offset, id)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1569, in read
    fd = self.get_fd(segment)
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1455, in get_fd
    fd = open_fd()
  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1436, in open_fd
    fd = open(self.segment_filename(segment), 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/srv/repos/[REDACTED]/repo/data/126/126862'
Borg server: Platform: Linux box-us7.borgbase.com 4.18.0-425.19.2.el8_7.x86_64 #1 SMP Fri Mar 17 01:52:38 EDT 2023 x86_64
Borg server: Linux: Unknown Linux
Borg server: Borg: 1.2.4  Python: CPython 3.9.13 msgpack: 1.0.3 fuse: None [pyfuse3,llfuse]
Borg server: PID: 1381721  CWD: /srv/repos/[REDACTED]
Borg server: sys.argv: ['/opt/venvs/borg-LATEST/bin/borg', 'serve', '--restrict-to-repository', '/srv/repos/[REDACTED]/repo', '--storage-quota', '100000M']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --debug'
Platform: Linux ae6e261dfd3d 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64
Linux: Unknown Linux
Borg: 1.2.4  Python: CPython 3.11.2 msgpack: 1.0.5 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 51  CWD: /
sys.argv: ['/usr/local/bin/borg', 'create', '--exclude-from', '/tmp/tmpngsnyhen', '--compression', 'auto,zstd', '--one-file-system', '--read-special', '--lock-wait', '60', '--list', '--filter', 'AMEx-', 'ssh://[REDACTED]::borgmatic-{now:%Y-%m-%dT%H:%M:%S.%f}', '/mnt/source', '/root/.borgmatic', '--stats', '--debug', '--show-rc']
SSH_ORIGINAL_COMMAND: None
terminating with error status, rc 2
Command 'borg create --exclude-from /tmp/tmpngsnyhen --compression auto,zstd --one-file-system --read-special --lock-wait 60 --list --filter AMEx- ssh://[REDACTED]::borgmatic-{now:%Y-%m-%dT%H:%M:%S.%f} /mnt/source /root/.borgmatic --stats --debug --show-rc' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

I decided to try borgmatic check, but it's also giving an error:

ae6e261dfd3d:/# borgmatic check -v 2
Ensuring legacy configuration is upgraded
borg --version --debug --show-rc
ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo: Running actions for repository
/etc/borgmatic.d/config.yaml: No commands to run for pre-actions hook
/etc/borgmatic.d/config.yaml: No commands to run for pre-check hook
ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo: Running consistency checks
borg info --lock-wait 60 --json ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo
Traceback (most recent call last):

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1453, in get_fd
    ts, fd = self.fds[segment]

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/lrucache.py", line 21, in __getitem__
    value = self._cache[key]  # raise KeyError if not found

KeyError: 126862


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/remote.py", line 240, in serve
    res = f(**args)

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1204, in get
    return self.io.read(segment, offset, id)

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1569, in read
    fd = self.get_fd(segment)

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1455, in get_fd
    fd = open_fd()

  File "/opt/venvs/borg-LATEST/lib/python3.9/site-packages/borg/repository.py", line 1436, in open_fd
    fd = open(self.segment_filename(segment), 'rb')

FileNotFoundError: [Errno 2] No such file or directory: '/srv/repos/[REDACTED]/repo/data/126/126862'

Borg server: Platform: Linux box-us7.borgbase.com 4.18.0-425.19.2.el8_7.x86_64 #1 SMP Fri Mar 17 01:52:38 EDT 2023 x86_64
Borg server: Linux: Unknown Linux  
Borg server: Borg: 1.2.4  Python: CPython 3.9.13 msgpack: 1.0.3 fuse: None [pyfuse3,llfuse]
Borg server: PID: 1381836  CWD: /srv/repos/[REDACTED]
Borg server: sys.argv: ['/opt/venvs/borg-LATEST/bin/borg', 'serve', '--restrict-to-repository', '/srv/repos/[REDACTED]/repo', '--storage-quota', '100000M']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve'
Platform: Linux ae6e261dfd3d 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64
Linux: Unknown Linux  
Borg: 1.2.4  Python: CPython 3.11.2 msgpack: 1.0.5 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 55  CWD: /
sys.argv: ['/usr/local/bin/borg', 'info', '--lock-wait', '60', '--json', 'ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo']
SSH_ORIGINAL_COMMAND: None

ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo: Error running actions for repository
Command '('borg', 'info', '--lock-wait', '60', '--json', 'ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo')' returned non-zero exit status 2.
/etc/borgmatic.d/config.yaml: No commands to run for on-error hook
/etc/borgmatic.d/config.yaml: An error occurred

summary:
/etc/borgmatic.d/config.yaml: An error occurred
ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo: Error running actions for repository
Command '('borg', 'info', '--lock-wait', '60', '--json', 'ssh://[REDACTED]@[REDACTED].repo.borgbase.com/./repo')' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

What should I do? I have already tried renaming the borg cache directory, which did not help.

[ThomasWaldmann]

FileNotFoundError: [Errno 2] No such file or directory: '/srv/repos/[REDACTED]/repo/data/126/126862'

Looks like you maybe have lost a file with some data.

You can try borg check --repair -v --progress ...

[ThomasWaldmann]

The repository part of the check didn't complain, so it seems that all the segment files still there were OK (crc32 check ok).

But, guess that one segment file was really lost somehow and contained all these archive metadata blocks it could not find any more while doing the (more high-level) archives part of the check.

So, the primary question is how that one segment file went missing, but guess that might be difficult or impossible to find out.

[ThomasWaldmann]

@m3nu can you check if disk / fs is ok?

[m3nu]

Yes, there was a pretty bad power equipment failure in one of the older US data centers we still have a few servers in and the file system of one machine needed repair despite running with a backup battery. Everyone affected was notified on the same day or via the usual monitoring alerts. Basically some data files touched just before the outage ended up in lost+found. Hence the error above.

As always, for BorgBase-specific issues, just email support@borgbase.com.

[BBaoVanC]

Yes, there was a pretty bad power equipment failure in one of the older US data centers we still have a few servers in and the file system of one machine needed repair despite running with a backup battery. Everyone affected was notified on the same day or via the usual monitoring alerts. Basically some data files touched just before the outage ended up in lost+found. Hence the error above.

As always, for BorgBase-specific issues, just email support@borgbase.com.

Hmm I did not get any notification apart from the automated notice from there not being backups for over 1 day (since they failed)...

I didn't think far enough ahead to make a copy of the repository in some way before running the repair... is it out of any chance of repair?

[m3nu]

Hard to tell without even knowing the repo ID or the server it actually used. Really best to just email to the address above.

[witten]

You know what, I bet it got stuck waiting for me to type YES on the repair (but I can't through borgmatic)

borgmatic switches to an interactive console (no logging capture) when using check --repair for exactly that reason—so you can type "YES" or "NO" if that prompt comes up.

The borgmatic traceback (I'm assuming resulting from ctrl-C?) just indicates it was waiting for output from another process (in this case Borg).

[BBaoVanC]

Hmm, it didn't even give me the prompt message saying to type YES when running through borgmatic (note I was using borgmatic borg check ... instead of borgmatic check ...)

[witten]

Ah, yes, borgmatic borg isn't smart enough to know whether to switch between interactive output or not, because it doesn't parse Borg's arguments (for the most part) and passes them directly to Borg. Instead, use borgmatic check. But if you ever want to drop down to run Borg directly, for instance for debugging purposes as suggested above, I'd recommend running borgmatic with --verbosity 2 and observing the exact Borg command it's invoking. Then you can always copy-and-paste it to run Borg manually. Hope that helps.

[BBaoVanC]

I'd recommend running borgmatic with --verbosity 2 and observing the exact Borg command it's invoking.

Yep that's what I ended up doing earlier, to do the repair :)

[ThomasWaldmann]

Closing this as outdated: it has seen no activity for a long time and predates the current borg releases.

If this still affects you on a current version - borg 1.4.x (stable) or the borg 2.0.0 beta - please open a new discussion with up-to-date details.