Skip to content

Commit c58fc2c

Browse files
committed
Add parsec erase_organization CLI command for the server
1 parent 74493a8 commit c58fc2c

14 files changed

Lines changed: 591 additions & 190 deletions

File tree

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
.. Parsec Cloud (https://parsec.cloud) Copyright (c) BUSL-1.1 2016-present Scille SAS
2+
3+
.. _doc_hosting_erase_organization:
4+
5+
Erase an organization
6+
=====================
7+
8+
Where & how the data are stored
9+
-------------------------------
10+
11+
For any given organization, data are split as follow:
12+
13+
- A PostgreSQL database containing the certificates (e.g. user/devices/workspaces) and
14+
encrypted workspaces metadata (e.g. content of folder, list of blocks for each file version).
15+
- A blockstore (e.g. S3) containing the blocks (i.e. encrypted pieces of data that compose the files).
16+
- On top of that, each Parsec client having access to the organization has an encrypted local database
17+
containing a copy of the certificates, metadata for the workspaces its has access to, and a subset
18+
of the blocks (depending on local cache configuration).
19+
20+
Erasing data
21+
------------
22+
23+
When no longer in use (or for legal reason) an organization can be erased from the Parsec server.
24+
25+
In practice this means:
26+
27+
1. Removing everything (certificates & metadata) related to the organization in the PostgreSQL database.
28+
2. Removing the blocks from the blockstore.
29+
3. Removing the remaining data from the clients.
30+
31+
Step 1: remove from PostgreSQL
32+
------------------------------
33+
34+
Erasing an organization from Parsec is both an uncommon and a (obviously !) a destructive operation.
35+
As such it is not available from the Administration API but instead must be triggered from the server CLI directly.
36+
37+
.. code-block:: bash
38+
39+
# On Parsec server
40+
parsec erase_organization --organization <OrgName> --db <database_url>
41+
42+
.. warning::
43+
44+
This operation cannot be undone. Make sure you have a backup of any data you may need before
45+
proceeding.
46+
47+
.. note::
48+
49+
After erasing an organization, it is possible to create a new organization with the same name
50+
since no trace of the previous one remains.
51+
52+
Step 2: Blockstore cleanup
53+
--------------------------
54+
55+
Once step 1 done, the blocks' decryption keys has been lost. In other words, everything
56+
stored in the blockstore and related to the organization is irrecoverable.
57+
Hence removing those data from the blockstore should be seen as an optional step to reclaim
58+
needlessly occupied space.
59+
60+
This can be done by manually removing from the blockstore the top level directory named after the organization.
61+
For example, if the organization was named ``CoolOrg``, remove the ``CoolOrg/`` prefix from the bucket.
62+
63+
.. note::
64+
65+
Blockstores have their own backup strategy. Typically AWS S3 allows for a bucket to
66+
have an history so that a data removal can be cancelled.
67+
You should pay attention to this to ensure the blocks have actually been removed.
68+
69+
Step 3: Clients cleanup
70+
-----------------------
71+
72+
From a Parsec client point of view, having the organization erased from the server is
73+
equivalent to being offline (i.e. the client cannot connect to the server).
74+
For this reason, an end-user is still able to use his Parsec client to work on the
75+
organization using the local cache (e.g. creating a new file in a workspace or
76+
reading an existing file that is in cache).
77+
78+
To prevent the users from accessing the local cache, the local configuration and data should be manually removed:
79+
80+
- Linux:
81+
- config: ``$XDG_DATA_HOME or $HOME/.local/share/parsec3/<device_id>`` (e.g. ``/home/alice/.local/share/parsec3/e68b7131394749a4bbd279bd087e6ae6``)
82+
- data: ``$XDG_CONFIG_HOME or $HOME/.config/parsec3/libparsec/devices/<device_id>`` (e.g. ``/home/alice/.config/parsec3/libparsec/devices/e68b7131394749a4bbd279bd087e6ae6``)
83+
- macOS:
84+
- config: ``$HOME/Library/Application Support/parsec3/<device_id>`` (e.g. ``/Users/Alice/Library/Application Support/parsec3/e68b7131394749a4bbd279bd087e6ae6``)
85+
- data: ``$HOME/Library/Application Support/parsec3/libparsec/devices/<device_id>`` (e.g. ``/Users/Alice/Library/Application Support/parsec3/libparsec/devices/e68b7131394749a4bbd279bd087e6ae6``)
86+
- Windows:
87+
- config: ``{FOLDERID_RoamingAppData}\parsec3\<device_id>`` (e.g. ``C:\Users\Alice\AppData\Roaming\parsec3\e68b7131394749a4bbd279bd087e6ae6``)
88+
- data: ``{FOLDERID_RoamingAppData}\parsec3\libparsec\devices\<device_id>`` (e.g. ``C:\Users\Alice\AppData\Roaming/parsec3/libparsec/devices/e68b7131394749a4bbd279bd087e6ae6``)

docs/hosting/administration/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,4 @@ Server Administration
1313
stats_organization
1414
freeze_users
1515
shared_recovery
16+
erase_organization

server/parsec/backend.py

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -156,9 +156,6 @@ async def test_customize_organization(
156156
skip_events_offset=len(template.events),
157157
)
158158

159-
async def test_drop_organization(self, id: OrganizationID) -> None:
160-
await self.organization.test_drop_organization(id)
161-
162159
async def test_load_template(self, template: TestbedTemplateContent) -> OrganizationID:
163160
org_id = OrganizationID(f"{template.id.title().replace('_', '')}OrgTemplate")
164161
match await self.organization.create(

server/parsec/cli/__init__.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88

99
import click
1010

11+
from parsec.cli.erase_organization import erase_organization
1112
from parsec.cli.export import export_realm
1213
from parsec.cli.export_email import export_email
1314
from parsec.cli.inspect import human_accesses
@@ -45,6 +46,7 @@ def cli() -> None:
4546
pass
4647

4748

49+
cli.add_command(erase_organization, "erase_organization")
4850
cli.add_command(run_cmd, "run")
4951
cli.add_command(migrate, "migrate")
5052
cli.add_command(export_realm, "export_realm")
Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# Parsec Cloud (https://parsec.cloud) Copyright (c) BUSL-1.1 2016-present Scille SAS
2+
from __future__ import annotations
3+
4+
import asyncio
5+
from typing import Any
6+
7+
import click
8+
9+
from parsec._parsec import (
10+
OrganizationID,
11+
)
12+
from parsec.cli.options import (
13+
db_server_options,
14+
debug_config_options,
15+
logging_config_options,
16+
)
17+
from parsec.cli.testbed import if_testbed_available
18+
from parsec.cli.utils import cli_exception_handler, spinner, start_backend
19+
from parsec.components.organization import OrganizationEraseBadOutcome
20+
from parsec.config import (
21+
BaseDatabaseConfig,
22+
DisabledBlockStoreConfig,
23+
LogLevel,
24+
MockedBlockStoreConfig,
25+
)
26+
27+
28+
class DevOption(click.Option):
29+
def handle_parse_result(
30+
self, ctx: click.Context, opts: Any, args: list[str]
31+
) -> tuple[Any, list[str]]:
32+
value, args = super().handle_parse_result(ctx, opts, args)
33+
if value:
34+
for key, value in (
35+
("debug", True),
36+
("db", "MOCKED"),
37+
("with_testbed", "coolorg"),
38+
("organization", "CoolorgOrgTemplate"),
39+
):
40+
if key not in opts:
41+
opts[key] = value
42+
43+
return value, args
44+
45+
46+
@click.command(short_help="Erase an organization from the database")
47+
@click.option("--organization", type=OrganizationID, help="Organization ID", required=True)
48+
@click.option("--yes", is_flag=True, help="Don't ask for confirmation before proceeding")
49+
@db_server_options
50+
# Add --log-level/--log-format/--log-file
51+
@logging_config_options(default_log_level="INFO")
52+
# Add --debug & --version
53+
@debug_config_options
54+
@if_testbed_available(
55+
click.option("--with-testbed", help="Start by populating with a testbed template")
56+
)
57+
@if_testbed_available(
58+
click.option(
59+
"--dev",
60+
cls=DevOption,
61+
is_flag=True,
62+
is_eager=True,
63+
help=(
64+
"Equivalent to `--debug --db=MOCKED --with-testbed=coolorg --organization CoolorgOrgTemplate`"
65+
),
66+
)
67+
)
68+
def erase_organization(
69+
organization: OrganizationID,
70+
db: BaseDatabaseConfig,
71+
db_max_connections: int,
72+
db_min_connections: int,
73+
log_level: LogLevel,
74+
log_format: str,
75+
log_file: str | None,
76+
yes: bool,
77+
debug: bool,
78+
with_testbed: str | None = None,
79+
dev: bool = False,
80+
) -> None:
81+
with cli_exception_handler(debug):
82+
asyncio.run(
83+
_erase_organization(
84+
yes=yes,
85+
db_config=db,
86+
debug=debug,
87+
with_testbed=with_testbed,
88+
organization_id=organization,
89+
)
90+
)
91+
92+
93+
async def _erase_organization(
94+
db_config: BaseDatabaseConfig,
95+
yes: bool,
96+
debug: bool,
97+
with_testbed: str | None,
98+
organization_id: OrganizationID,
99+
) -> None:
100+
# Can use a dummy blockstore config since we are not going to query it
101+
if with_testbed is None:
102+
blockstore_config = DisabledBlockStoreConfig()
103+
else:
104+
# Testbed template might need to create some blocks
105+
blockstore_config = MockedBlockStoreConfig()
106+
107+
display_org = click.style(organization_id.str, fg="yellow")
108+
click.echo(
109+
f"You are about to entirely erase the {display_org} organization from the database, this action cannot be undone."
110+
)
111+
112+
display_bucket_path = click.style(f"{organization_id.str}/", fg="yellow")
113+
click.echo("Notes:")
114+
click.echo(
115+
"- No trace of the organization will remain, so it will be possible to re-create another organization with the same name."
116+
)
117+
click.echo(
118+
f"- The organization's blocks won't be erased from the blockstore, you should instead manually remove the {display_bucket_path} top level directory from it."
119+
)
120+
click.echo("")
121+
122+
if not yes:
123+
confirmation = click.prompt("To confirm, type the name of the organization")
124+
if confirmation != organization_id.str:
125+
raise RuntimeError("Organization name does not match, aborting")
126+
127+
async with start_backend(
128+
db_config=db_config,
129+
blockstore_config=blockstore_config,
130+
debug=debug,
131+
populate_with_template=with_testbed,
132+
) as backend:
133+
async with spinner("Removing from database..."):
134+
outcome = await backend.organization.erase(id=organization_id)
135+
match outcome:
136+
case None:
137+
pass
138+
case OrganizationEraseBadOutcome.ORGANIZATION_NOT_FOUND:
139+
raise RuntimeError("Organization doesn't exist")

server/parsec/cli/testbed.py

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -179,9 +179,10 @@ async def customize_organization(self, id: OrganizationID, customization: bytes)
179179
cooked_customization = testbed.test_load_testbed_customization(template, customization) # pyright: ignore [reportPossiblyUnboundVariable]
180180
await self.backend.test_customize_organization(id, template, cooked_customization)
181181

182-
async def drop_organization(self, id: OrganizationID) -> None:
183-
await self.backend.test_drop_organization(id)
184-
del self.template_per_org[id]
182+
async def drop_organization_idempotent(self, id: OrganizationID) -> None:
183+
# Ignore errors (in case the organization doesn't exist) to be idempotent
184+
await self.backend.organization.erase(id)
185+
self.template_per_org.pop(id, None)
185186

186187

187188
testbed_router = APIRouter(tags=["testbed"])
@@ -244,8 +245,7 @@ async def test_new(template: str, request: Request, background_tasks: Background
244245
async def _organization_garbage_collector():
245246
await asyncio.sleep(orga_life_limit)
246247
logger.info("Dropping testbed org due to time limit", organization=new_org_id.str)
247-
# Dropping is idempotent, so no need for error handling
248-
await testbed.backend.test_drop_organization(new_org_id)
248+
await testbed.drop_organization_idempotent(new_org_id)
249249

250250
background_tasks.add_task(_organization_garbage_collector)
251251

@@ -279,8 +279,7 @@ async def test_drop(raw_organization_id: str, request: Request) -> Response:
279279
except ValueError:
280280
return Response(status_code=400, content=b"")
281281

282-
# Dropping is idempotent, so no need for error handling
283-
await testbed.drop_organization(organization_id)
282+
await testbed.drop_organization_idempotent(organization_id)
284283

285284
return Response(status_code=200, content=b"")
286285

server/parsec/components/memory/organization.py

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
OrganizationCreateBadOutcome,
3232
OrganizationDump,
3333
OrganizationDumpTopics,
34+
OrganizationEraseBadOutcome,
3435
OrganizationGetBadOutcome,
3536
OrganizationGetTosBadOutcome,
3637
OrganizationStats,
@@ -383,6 +384,16 @@ async def update(
383384
if tos is not Unset:
384385
await self._event_bus.send(EventOrganizationTosUpdated(organization_id=id))
385386

387+
@override
388+
async def erase(
389+
self,
390+
id: OrganizationID,
391+
) -> None | OrganizationEraseBadOutcome:
392+
try:
393+
del self._data.organizations[id]
394+
except KeyError:
395+
return OrganizationEraseBadOutcome.ORGANIZATION_NOT_FOUND
396+
386397
@override
387398
async def get_tos(
388399
self, id: OrganizationID
@@ -437,10 +448,6 @@ async def test_dump_topics(self, id: OrganizationID) -> OrganizationDumpTopics:
437448
shamir_recovery=org.per_topic_last_timestamp.get("shamir_recovery"),
438449
)
439450

440-
@override
441-
async def test_drop_organization(self, id: OrganizationID) -> None:
442-
self._data.organizations.pop(id, None)
443-
444451
@override
445452
async def test_duplicate_organization(
446453
self, source_id: OrganizationID, target_id: OrganizationID

server/parsec/components/organization.py

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,10 @@ class OrganizationUpdateBadOutcome(BadOutcomeEnum):
227227
ORGANIZATION_NOT_FOUND = auto()
228228

229229

230+
class OrganizationEraseBadOutcome(BadOutcomeEnum):
231+
ORGANIZATION_NOT_FOUND = auto()
232+
233+
230234
@dataclass(slots=True)
231235
class OrganizationDumpTopics:
232236
common: DateTime
@@ -316,6 +320,12 @@ async def update(
316320
) -> None | OrganizationUpdateBadOutcome:
317321
raise NotImplementedError
318322

323+
async def erase(
324+
self,
325+
id: OrganizationID,
326+
) -> None | OrganizationEraseBadOutcome:
327+
raise NotImplementedError
328+
319329
async def get_tos(
320330
self, id: OrganizationID
321331
) -> TermsOfService | None | OrganizationGetTosBadOutcome:
@@ -329,9 +339,6 @@ async def test_dump_organizations(
329339
async def test_dump_topics(self, id: OrganizationID) -> OrganizationDumpTopics:
330340
raise NotImplementedError
331341

332-
async def test_drop_organization(self, id: OrganizationID) -> None:
333-
raise NotImplementedError
334-
335342
async def test_duplicate_organization(
336343
self, source_id: OrganizationID, target_id: OrganizationID
337344
) -> None:

0 commit comments

Comments
 (0)