Skip to content

Commit 5d302e4

Browse files
authored
Delta client (#6589)
## 📖 Description Adds experimental `deltaIndex` feature that enables the client to use delta+baseline when it is available from the source. During add, we attempt to find the delta package at a fixed name, then the baseline it references. If this is all successful, the source data has a new JSON representation to record the multiple pieces of information. When updating or opening the source, if we believe that delta is available (because we have both the delta and baseline identities), only the delta is acceptable. Any failure should escape to be reported and fixed. We may consider fallback in the future, but it is much more complex than it would first appear due to the baseline sharing identity with the full index.
1 parent da1379b commit 5d302e4

50 files changed

Lines changed: 3557 additions & 123 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎doc/Settings.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -465,3 +465,15 @@ Enables numbered choices for ambiguous single-package `install`, `show`, and `do
465465
```
466466

467467
`--disable-interactivity` and redirected input or output still prevent prompting.
468+
469+
### deltaIndex
470+
471+
This feature changes how the client acquires updates to a pre-indexed source. Instead of downloading the whole index every time, it downloads a delta describing only what has changed since a periodically published baseline, and combines the two locally. The baseline is downloaded only when the source does not already have the one the delta was built against.
472+
473+
The data presented by the source is the same either way; only how it is acquired differs. A source that does not publish a delta is unaffected, and the client falls back to downloading the full index.
474+
475+
```json
476+
"experimentalFeatures": {
477+
"deltaIndex": true
478+
},
479+
```

‎schemas/JSON/settings/settings.schema.0.2.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -348,6 +348,11 @@
348348
"description": "Enable support for some commands to resume",
349349
"type": "boolean",
350350
"default": false
351+
},
352+
"deltaIndex": {
353+
"description": "Enable acquiring source index updates as a delta against a baseline, rather than downloading the full index",
354+
"type": "boolean",
355+
"default": false
351356
}
352357
}
353358
},

‎src/AppInstallerCLICore/Command.h‎

Lines changed: 1 addition & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,11 @@
55
#include "ExecutionContext.h"
66
#include "Invocation.h"
77
#include "Resources.h"
8+
#include <AppInstallerLanguageUtilities.h>
89
#include <winget/UserSettings.h>
910
#include <winget/ExperimentalFeature.h>
1011
#include <winget/GroupPolicy.h>
1112

12-
#include <initializer_list>
1313
#include <memory>
1414
#include <optional>
1515
#include <ostream>
@@ -138,20 +138,6 @@ namespace AppInstaller::CLI
138138
std::string m_commandArguments;
139139
};
140140

141-
template <typename Container>
142-
Container InitializeFromMoveOnly(std::initializer_list<typename Container::value_type> il)
143-
{
144-
using Value = typename Container::value_type;
145-
Container result;
146-
147-
for (const auto& v : il)
148-
{
149-
result.emplace_back(std::move(*const_cast<Value*>(&v)));
150-
}
151-
152-
return result;
153-
}
154-
155141
void ExecuteWithoutLoggingSuccess(Execution::Context& context, Command* command);
156142

157143
int Execute(Execution::Context& context, std::unique_ptr<Command>& command);

‎src/AppInstallerCLIE2ETests/Constants.cs‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,29 @@ public class Constants
5353
public const string RestTestSourceUrl = @"https://localhost:5001/TestKit/TestData/TestRestSource";
5454
public const string RestTestSourceType = "Microsoft.Rest";
5555

56+
// The delta capable test source. It is published separately from the source above so that a
57+
// client that knows nothing about deltas keeps seeing exactly what it saw before.
58+
public const string DeltaTestSourceName = @"TestDeltaSource";
59+
public const string DeltaTestSourceDirectory = @"deltaSource";
60+
public const string DeltaTestSourceUrl = @"https://localhost:5001/TestKit/" + DeltaTestSourceDirectory;
61+
public const string DeltaTestSourceIdentityName = @"WingetE2EDelta.Tests";
62+
public const string DeltaTestSourceDeltaIdentityName = @"WingetE2EDelta.Tests.Delta";
63+
public const string DeltaTestSourceBaselineIdentityName = @"WingetE2EDelta.Tests.Baseline";
64+
public const string DeltaTestSourceIdentifier = DeltaTestSourceIdentityName + "_" + AICLIPackagePublisherHash;
65+
public const string DeltaTestSourceDeltaIdentifier = DeltaTestSourceDeltaIdentityName + "_" + AICLIPackagePublisherHash;
66+
public const string DeltaTestSourceBaselineIdentifier = DeltaTestSourceBaselineIdentityName + "_" + AICLIPackagePublisherHash;
67+
68+
// The baseline is published older than the full index, because a baseline is captured once
69+
// and then described by a succession of deltas.
70+
public const string DeltaTestSourceVersion = @"2020.805.713.335";
71+
public const string DeltaTestSourceBaselineVersion = @"2020.805.713.334";
72+
public const string DeltaTestSourceBaselineRelativePath = @"baseline/source2.msix";
73+
74+
// Left out of the baseline, so that it exists only in the delta. Finding it is what tells a
75+
// merged result apart from either half read on its own.
76+
public const string DeltaOnlyManifestFileName = @"TestExampleInstaller.yaml";
77+
public const string DeltaOnlyPackageIdentifier = @"AppInstallerTest.TestExampleInstaller";
78+
5679
public const string AICLIPackageFamilyName = "WinGetDevCLI_8wekyb3d8bbwe";
5780
public const string AICLIPackageName = "WinGetDevCLI";
5881
public const string AICLIPackagePublisherHash = "8wekyb3d8bbwe";
Lines changed: 247 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,247 @@
1+
// -----------------------------------------------------------------------------
2+
// <copyright file="DeltaSource.cs" company="Microsoft Corporation">
3+
// Copyright (c) Microsoft Corporation. Licensed under the MIT License.
4+
// </copyright>
5+
// -----------------------------------------------------------------------------
6+
7+
namespace AppInstallerCLIE2ETests
8+
{
9+
using AppInstallerCLIE2ETests.Helpers;
10+
using NUnit.Framework;
11+
12+
/// <summary>
13+
/// Tests the delta index against the deployed package mechanism.
14+
///
15+
/// Unit tests cover the shape of a delta source and the merging of a delta with its baseline,
16+
/// but they run unpackaged, so they can only ever reach the local file store. Everything here
17+
/// is about the half they structurally cannot touch: deploying the packages, keeping three
18+
/// separate deployed identities apart, and the clean up that follows a removal.
19+
/// </summary>
20+
public class DeltaSource : BaseCommand
21+
{
22+
/// <summary>
23+
/// The experimental feature, as it is named in the settings file.
24+
/// </summary>
25+
private const string DeltaFeatureName = "deltaIndex";
26+
27+
/// <summary>
28+
/// A package that is in the baseline, and so is not one the delta has to supply.
29+
/// </summary>
30+
private const string BaselinePackageIdentifier = Constants.PortableExePackageId;
31+
32+
/// <summary>
33+
/// One time set up.
34+
/// </summary>
35+
[OneTimeSetUp]
36+
public void OneTimeSetup()
37+
{
38+
WinGetSettingsHelper.ConfigureFeature(DeltaFeatureName, true);
39+
}
40+
41+
/// <summary>
42+
/// One time tear down.
43+
/// </summary>
44+
[OneTimeTearDown]
45+
public void OneTimeTeardown()
46+
{
47+
RemoveDeltaSource();
48+
WinGetSettingsHelper.ConfigureFeature(DeltaFeatureName, false);
49+
}
50+
51+
/// <summary>
52+
/// Test set up.
53+
/// </summary>
54+
[SetUp]
55+
public void Setup()
56+
{
57+
RemoveDeltaSource();
58+
WinGetSettingsHelper.ConfigureFeature(DeltaFeatureName, true);
59+
}
60+
61+
/// <summary>
62+
/// Adding a delta capable source acquires the delta and the baseline rather than the full
63+
/// index, and what it then presents is the two of them merged.
64+
/// </summary>
65+
[Test]
66+
public void DeltaSourceAdd()
67+
{
68+
AddDeltaSource();
69+
70+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceDeltaIdentityName), Is.True, "The delta package should be deployed");
71+
Assert.That(
72+
GetDeployedPackageVersion(Constants.DeltaTestSourceBaselineIdentityName),
73+
Is.EqualTo(Constants.DeltaTestSourceBaselineVersion),
74+
"The baseline should have been deployed at its own version");
75+
76+
// The baseline is a package of its own, so a source on the delta holds no full index at
77+
// all. Were the two to share an identity this could not be asked.
78+
Assert.That(
79+
IsPackageDeployed(Constants.DeltaTestSourceIdentityName),
80+
Is.False,
81+
"The full index should not have been acquired");
82+
83+
// This package was left out of the baseline, so it exists only in the delta.
84+
RequireFound(Constants.DeltaOnlyPackageIdentifier);
85+
86+
// And a package that is in the baseline is still there, so the delta did not displace it.
87+
RequireFound(BaselinePackageIdentifier);
88+
}
89+
90+
/// <summary>
91+
/// A source added while the feature was off holds the full index, and goes on holding it.
92+
///
93+
/// Which packages a source is made of is established when it is added and recorded in its
94+
/// details; an update reads those details and never probes for anything else. So turning
95+
/// the feature on does not move an existing source onto the delta, and the source has to be
96+
/// re-added for that. This is deliberate -- see the no-migration decision -- and it is
97+
/// asserted here because nothing about it is obvious from the outside.
98+
/// </summary>
99+
[Test]
100+
public void DeltaSourceNotAdoptedByExistingFullIndexSource()
101+
{
102+
WinGetSettingsHelper.ConfigureFeature(DeltaFeatureName, false);
103+
104+
AddDeltaSource();
105+
106+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceDeltaIdentityName), Is.False, "No delta should be involved while the feature is off");
107+
Assert.That(
108+
GetDeployedPackageVersion(Constants.DeltaTestSourceIdentityName),
109+
Is.EqualTo(Constants.DeltaTestSourceVersion),
110+
"The full index should be deployed at its own version");
111+
112+
WinGetSettingsHelper.ConfigureFeature(DeltaFeatureName, true);
113+
114+
var updateResult = TestCommon.RunAICLICommand("source update", $"-n {Constants.DeltaTestSourceName}");
115+
Assert.That(updateResult.ExitCode, Is.EqualTo(Constants.ErrorCode.S_OK));
116+
Assert.That(updateResult.StdOut, Does.Contain("Done"));
117+
118+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceDeltaIdentityName), Is.False, "The update should not have acquired a delta");
119+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceBaselineIdentityName), Is.False, "The update should not have acquired a baseline");
120+
Assert.That(
121+
GetDeployedPackageVersion(Constants.DeltaTestSourceIdentityName),
122+
Is.EqualTo(Constants.DeltaTestSourceVersion),
123+
"The source should still be served by the full index");
124+
125+
// The full index holds everything, so what the source presents is unchanged.
126+
RequireFound(Constants.DeltaOnlyPackageIdentifier);
127+
RequireFound(BaselinePackageIdentifier);
128+
}
129+
130+
/// <summary>
131+
/// Disabling delta after adding a source acquires the full index and keeps it usable.
132+
/// </summary>
133+
/// <param name="updateSource">Whether to explicitly update before searching.</param>
134+
[TestCase(false)]
135+
[TestCase(true)]
136+
public void DeltaSourceFeatureDisabled(bool updateSource)
137+
{
138+
AddDeltaSource();
139+
140+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceDeltaIdentityName), Is.True);
141+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceBaselineIdentityName), Is.True);
142+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceIdentityName), Is.False);
143+
RequireFound(Constants.DeltaOnlyPackageIdentifier);
144+
RequireFound(BaselinePackageIdentifier);
145+
146+
WinGetSettingsHelper.ConfigureFeature(DeltaFeatureName, false);
147+
148+
if (updateSource)
149+
{
150+
var updateResult = TestCommon.RunAICLICommand("source update", $"-n {Constants.DeltaTestSourceName}");
151+
Assert.That(updateResult.ExitCode, Is.EqualTo(Constants.ErrorCode.S_OK));
152+
Assert.That(updateResult.StdOut, Does.Contain("Done"));
153+
}
154+
155+
RequireFound(Constants.DeltaOnlyPackageIdentifier);
156+
RequireFound(BaselinePackageIdentifier);
157+
Assert.That(
158+
GetDeployedPackageVersion(Constants.DeltaTestSourceIdentityName),
159+
Is.EqualTo(Constants.DeltaTestSourceVersion),
160+
"The source should be served by the full index with delta disabled");
161+
162+
var removeResult = TestCommon.RunAICLICommand("source remove", $"-n {Constants.DeltaTestSourceName}");
163+
Assert.That(removeResult.ExitCode, Is.EqualTo(Constants.ErrorCode.S_OK));
164+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceIdentityName), Is.False);
165+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceDeltaIdentityName), Is.False);
166+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceBaselineIdentityName), Is.False);
167+
}
168+
169+
/// <summary>
170+
/// Removing the source takes both deployed packages with it. A delta source leaves more
171+
/// behind than an ordinary one, so there is more to get wrong.
172+
/// </summary>
173+
[Test]
174+
public void DeltaSourceRemove()
175+
{
176+
AddDeltaSource();
177+
178+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceDeltaIdentityName), Is.True);
179+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceBaselineIdentityName), Is.True);
180+
181+
var removeResult = TestCommon.RunAICLICommand("source remove", $"-n {Constants.DeltaTestSourceName}");
182+
Assert.That(removeResult.ExitCode, Is.EqualTo(Constants.ErrorCode.S_OK));
183+
Assert.That(removeResult.StdOut, Does.Contain("Done"));
184+
185+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceDeltaIdentityName), Is.False, "The delta package should have been removed");
186+
Assert.That(IsPackageDeployed(Constants.DeltaTestSourceBaselineIdentityName), Is.False, "The baseline package should have been removed");
187+
188+
var listResult = TestCommon.RunAICLICommand("source list", $"-n {Constants.DeltaTestSourceName}");
189+
Assert.That(listResult.ExitCode, Is.EqualTo(Constants.ErrorCode.ERROR_SOURCE_NAME_DOES_NOT_EXIST));
190+
}
191+
192+
/// <summary>
193+
/// Adds the delta source, requiring that it succeed.
194+
/// </summary>
195+
private static void AddDeltaSource()
196+
{
197+
var result = TestCommon.RunAICLICommand("source add", $"{Constants.DeltaTestSourceName} {Constants.DeltaTestSourceUrl} --trust-level trusted");
198+
Assert.That(result.ExitCode, Is.EqualTo(Constants.ErrorCode.S_OK));
199+
Assert.That(result.StdOut, Does.Contain("Done"));
200+
}
201+
202+
/// <summary>
203+
/// Removes the delta source, tolerating its not having been added.
204+
/// </summary>
205+
private static void RemoveDeltaSource()
206+
{
207+
TestCommon.RunAICLICommand("source remove", $"-n {Constants.DeltaTestSourceName}");
208+
}
209+
210+
/// <summary>
211+
/// Requires that the delta source presents the given package.
212+
/// </summary>
213+
/// <param name="packageIdentifier">The package identifier.</param>
214+
private static void RequireFound(string packageIdentifier)
215+
{
216+
var result = TestCommon.RunAICLICommand("search", $"{packageIdentifier} --source {Constants.DeltaTestSourceName}");
217+
Assert.That(result.ExitCode, Is.EqualTo(Constants.ErrorCode.S_OK), $"{packageIdentifier} should be found");
218+
Assert.That(result.StdOut, Does.Contain(packageIdentifier));
219+
}
220+
221+
/// <summary>
222+
/// Whether a package with the given identity name is deployed for the current user.
223+
/// </summary>
224+
/// <param name="identityName">The package identity name.</param>
225+
/// <returns>True if it is deployed.</returns>
226+
private static bool IsPackageDeployed(string identityName)
227+
{
228+
return !string.IsNullOrEmpty(GetDeployedPackageVersion(identityName));
229+
}
230+
231+
/// <summary>
232+
/// Gets the version of a deployed package, or an empty string when it is not deployed.
233+
/// </summary>
234+
/// <param name="identityName">The package identity name.</param>
235+
/// <returns>The version, or an empty string.</returns>
236+
private static string GetDeployedPackageVersion(string identityName)
237+
{
238+
// The name has to match exactly, which matters here because the delta's identity name
239+
// has the full index's as a prefix.
240+
var result = TestCommon.RunCommandWithResult(
241+
"powershell",
242+
$"-Command \"(Get-AppxPackage | Where-Object {{ $_.Name -ceq '{identityName}' }} | Select-Object -First 1).Version\"");
243+
244+
return result.StdOut.Trim();
245+
}
246+
}
247+
}

‎src/AppInstallerCLIE2ETests/Helpers/TestIndex.cs‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -192,6 +192,20 @@ public static void GenerateE2ESource()
192192
{
193193
CertFile = testParams.PackageCertificatePath,
194194
},
195+
Delta = new ()
196+
{
197+
RelativeDirectory = Constants.DeltaTestSourceDirectory,
198+
IdentityName = Constants.DeltaTestSourceIdentityName,
199+
DeltaIdentityName = Constants.DeltaTestSourceDeltaIdentityName,
200+
BaselineIdentityName = Constants.DeltaTestSourceBaselineIdentityName,
201+
Version = Constants.DeltaTestSourceVersion,
202+
BaselineVersion = Constants.DeltaTestSourceBaselineVersion,
203+
BaselineRelativePath = Constants.DeltaTestSourceBaselineRelativePath,
204+
ManifestsNotInBaseline = new ()
205+
{
206+
Constants.DeltaOnlyManifestFileName,
207+
},
208+
},
195209
};
196210

197211
WinGetLocalSource.CreateLocalSource(e2eSource);

‎src/AppInstallerCLIE2ETests/TestData/localsource.json‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,5 +53,17 @@
5353
"Signature": {
5454
"CertFile": "%TestSigningCert_PfxPath%",
5555
"Password": "%TestSigningCert_Password%"
56+
},
57+
"Delta": {
58+
"RelativeDirectory": "deltaSource",
59+
"IdentityName": "WingetE2EDelta.Tests",
60+
"DeltaIdentityName": "WingetE2EDelta.Tests.Delta",
61+
"BaselineIdentityName": "WingetE2EDelta.Tests.Baseline",
62+
"Version": "2020.805.713.335",
63+
"BaselineVersion": "2020.805.713.334",
64+
"BaselineRelativePath": "baseline/source2.msix",
65+
"ManifestsNotInBaseline": [
66+
"TestExampleInstaller.yaml"
67+
]
5668
}
5769
}

‎src/AppInstallerCLITests/AppInstallerCLITests.vcxproj‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,7 @@
218218
<ClInclude Include="TestConfiguration.h" />
219219
<ClInclude Include="TestRestRequestHandler.h" />
220220
<ClInclude Include="TestHooks.h" />
221+
<ClInclude Include="TestMsixPackage.h" />
221222
<ClInclude Include="TestSettings.h" />
222223
<ClInclude Include="TestSource.h" />
223224
<ClInclude Include="WorkflowCommon.h" />
@@ -338,6 +339,7 @@
338339
<ClCompile Include="TableOutput.cpp" />
339340
<ClCompile Include="TestCertificates.cpp" />
340341
<ClCompile Include="TestCommon.cpp" />
342+
<ClCompile Include="TestMsixPackage.cpp" />
341343
<ClCompile Include="WorkflowCommon.cpp" />
342344
<ClCompile Include="WorkflowGroupPolicy.cpp" />
343345
<ClCompile Include="Yaml.cpp" />

0 commit comments

Comments
 (0)