Skip to content

Commit 08e467f

Browse files
HarryCordewenerclaude
HarryCordewener
and
claude
authored
Data markup: layers that say what text is, kept by WithoutVisualMarkup() (#67)
* Add data markup: layers no format draws, kept by WithoutVisualMarkup() IDataMarkup marks a layer that says what text is rather than how it looks, such as a list separator. The renderer takes data layers off each run before any emitter sees it; WithoutVisualMarkup() strips every other layer and keeps them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LyLns5DC45YxSeQRcmwD6r * Add MarkupText.WithoutDataMarkup(): the text with its data layers taken off Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LyLns5DC45YxSeQRcmwD6r --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent d40a753 commit 08e467f

8 files changed

Lines changed: 326 additions & 2 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,20 @@ and `MarkupString.Pueblo`. The packages share one version and are released toget
66
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project
77
follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
88

9-
## 2.20.0 — Unreleased
9+
## 2.21.0 — Unreleased
10+
11+
### Added
12+
13+
- **Data markup.** `IDataMarkup` marks a layer that says what the text is rather than how it looks,
14+
such as a list separator. No format draws it: the renderer takes it off each run before any emitter
15+
sees the run, so the text is written as the layers around it write it. It is stored, carried and
16+
compared like any other layer. See [Data markup](docs/custom-markup.md#data-markup).
17+
- **`MarkupText.WithoutVisualMarkup()`**: the plain text with every data layer kept, for stripping
18+
colour from text that also carries data.
19+
- **`MarkupText.WithoutDataMarkup()`**: the text with every data layer taken off and its visual
20+
markup and points kept, for text whose data must not be trusted.
21+
22+
## 2.20.0
1023

1124
### Added
1225

‎MarkupString.Tests/DataMarkupTests.cs‎

Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
using System.Text.Json;
2+
using MarkupString.Ansi;
3+
using MarkupString.Html;
4+
namespace MarkupString.Tests;
5+
6+
/// <summary>
7+
/// Data markup says what text is, not how it looks: no format draws it, taking the visual markup off keeps
8+
/// it, and it is carried and stored like any other layer.
9+
/// </summary>
10+
public class DataMarkupTests
11+
{
12+
private sealed record Separator : IDataMarkup;
13+
14+
private sealed class SeparatorCodec : IMarkupCodec
15+
{
16+
public string Kind => "test-separator";
17+
public Type MarkupType => typeof(Separator);
18+
public void Write(Utf8JsonWriter writer, IMarkup markup) { }
19+
public IMarkup Read(JsonElement element) => new Separator();
20+
}
21+
22+
/// <summary>Draws a separator as <c>{|}</c>, to show that the renderer never calls an emitter for data.</summary>
23+
private sealed class SeparatorEmitter : IMarkupEmitter
24+
{
25+
public Type MarkupType => typeof(Separator);
26+
public MarkupFormat Format => MarkupFormat.Ansi;
27+
public void Emit(IMarkup markup, ReadOnlySpan<char> body, in EmitContext context, System.Buffers.IBufferWriter<char> output)
28+
{
29+
output.Write("{");
30+
output.Write(body);
31+
output.Write("}");
32+
}
33+
}
34+
35+
private static readonly MarkupRegistry Registry = MarkupRegistry.Empty.WithAnsi().WithHtml().With(new SeparatorCodec());
36+
37+
private static readonly AnsiMarkup Red = AnsiMarkup.Create(foreground: new AnsiColor.Standard(1, false));
38+
39+
private static MarkupText Marked(string text) => MarkupText.Wrap(new Separator(), text);
40+
41+
private static bool IsData(MarkupText text, int index) =>
42+
text.Runs.Any(run => index >= run.Start && index < run.End && run.Markups.Any(markup => markup is IDataMarkup));
43+
44+
[Test]
45+
public async Task DataIsWrittenAsItsText()
46+
{
47+
var text = MarkupText.Concat([MarkupText.Plain("a"), Marked("|"), MarkupText.Plain("b")]);
48+
49+
await Assert.That(text.Render(MarkupFormat.Ansi, Registry)).IsEqualTo("a|b");
50+
await Assert.That(text.Render(MarkupFormat.Html, Registry)).IsEqualTo("a|b");
51+
await Assert.That(text.Render(MarkupFormat.Plain, Registry)).IsEqualTo("a|b");
52+
}
53+
54+
[Test]
55+
public async Task DataUnderColourIsWrittenInTheColour()
56+
{
57+
var coloured = MarkupText.Concat([MarkupText.Plain("a"), MarkupText.Wrap(Red, Marked("|")), MarkupText.Plain("b")]);
58+
var colourOnly = MarkupText.Concat([MarkupText.Plain("a"), MarkupText.Wrap(Red, "|"), MarkupText.Plain("b")]);
59+
60+
await Assert.That(coloured.Render(MarkupFormat.Ansi, Registry)).IsEqualTo(colourOnly.Render(MarkupFormat.Ansi, Registry));
61+
await Assert.That(coloured.Render(MarkupFormat.Html, Registry)).IsEqualTo(colourOnly.Render(MarkupFormat.Html, Registry));
62+
}
63+
64+
[Test]
65+
public async Task AnEmitterForDataIsNeverCalled()
66+
{
67+
var registry = Registry.With(new SeparatorEmitter());
68+
69+
await Assert.That(MarkupText.Concat([MarkupText.Plain("a"), Marked("|")]).Render(MarkupFormat.Ansi, registry)).IsEqualTo("a|");
70+
}
71+
72+
[Test]
73+
public async Task WithoutVisualMarkup_KeepsTheDataAndDropsTheColour()
74+
{
75+
var text = MarkupText.Concat([MarkupText.Wrap(Red, "a"), MarkupText.Wrap(Red, Marked("|")), MarkupText.Plain("b"), Marked("|")]);
76+
77+
var stripped = text.WithoutVisualMarkup();
78+
79+
await Assert.That(stripped.Text).IsEqualTo("a|b|");
80+
await Assert.That(stripped.Runs.Length).IsEqualTo(2);
81+
await Assert.That(stripped.Runs.All(run => run.Markups.Count == 1 && run.Markups[0] is Separator)).IsTrue();
82+
await Assert.That(IsData(stripped, 1)).IsTrue();
83+
await Assert.That(IsData(stripped, 3)).IsTrue();
84+
await Assert.That(IsData(stripped, 0)).IsFalse();
85+
}
86+
87+
[Test]
88+
public async Task WithoutVisualMarkup_JoinsDataThatColourHadSplit()
89+
{
90+
var text = MarkupText.Concat([MarkupText.Wrap(Red, Marked("|")), Marked("|")]);
91+
92+
var stripped = text.WithoutVisualMarkup();
93+
94+
await Assert.That(stripped.Runs.Length).IsEqualTo(1);
95+
await Assert.That(stripped.Runs[0].Length).IsEqualTo(2);
96+
}
97+
98+
[Test]
99+
public async Task WithoutVisualMarkup_WithNoData_IsThePlainText()
100+
{
101+
var text = MarkupText.Concat([MarkupText.Wrap(Red, "a"), MarkupText.Bell(), MarkupText.Plain("b")]);
102+
103+
var stripped = text.WithoutVisualMarkup();
104+
105+
await Assert.That(stripped.Text).IsEqualTo("ab");
106+
await Assert.That(stripped.Runs.IsDefaultOrEmpty).IsTrue();
107+
}
108+
109+
[Test]
110+
public async Task WithoutVisualMarkup_DropsPointsWithTheirCarriers()
111+
{
112+
var text = MarkupText.Concat([Marked("|"), MarkupText.Bell(), MarkupText.Plain("b")]);
113+
114+
var stripped = text.WithoutVisualMarkup();
115+
116+
await Assert.That(stripped.Text).IsEqualTo("|b");
117+
await Assert.That(IsData(stripped, 0)).IsTrue();
118+
}
119+
120+
[Test]
121+
public async Task WithoutDataMarkup_KeepsTheColourAndDropsTheData()
122+
{
123+
var text = MarkupText.Concat([MarkupText.Wrap(Red, "a"), MarkupText.Wrap(Red, Marked("|")), MarkupText.Plain("b"), Marked("|")]);
124+
125+
var unmarked = text.WithoutDataMarkup();
126+
127+
await Assert.That(unmarked.Text).IsEqualTo("a|b|");
128+
await Assert.That(IsData(unmarked, 1)).IsFalse();
129+
await Assert.That(IsData(unmarked, 3)).IsFalse();
130+
await Assert.That(unmarked.Render(MarkupFormat.Ansi, Registry))
131+
.IsEqualTo(MarkupText.Concat([MarkupText.Wrap(Red, "a|"), MarkupText.Plain("b|")]).Render(MarkupFormat.Ansi, Registry));
132+
}
133+
134+
[Test]
135+
public async Task WithoutDataMarkup_KeepsPoints()
136+
{
137+
var text = MarkupText.Concat([Marked("|"), MarkupText.Bell(), MarkupText.Plain("b")]);
138+
139+
var unmarked = text.WithoutDataMarkup();
140+
141+
await Assert.That(unmarked.Text).IsEqualTo(text.Text);
142+
await Assert.That(unmarked.ToPlainText()).IsEqualTo("|b");
143+
await Assert.That(IsData(unmarked, 0)).IsFalse();
144+
}
145+
146+
[Test]
147+
public async Task WithoutDataMarkup_WithNoData_IsTheSameText()
148+
{
149+
var text = MarkupText.Concat([MarkupText.Wrap(Red, "a"), MarkupText.Plain("b")]);
150+
151+
await Assert.That(ReferenceEquals(text.WithoutDataMarkup(), text)).IsTrue();
152+
}
153+
154+
[Test]
155+
public async Task DataIsIgnoredByEquality()
156+
=> await Assert.That(MarkupText.Concat([MarkupText.Plain("a"), Marked("|")])).IsEqualTo(MarkupText.Plain("a|"));
157+
158+
[Test]
159+
public async Task DataIsStoredAndReadBack()
160+
{
161+
var text = MarkupText.Concat([MarkupText.Plain("a"), Marked("|"), MarkupText.Plain("b")]);
162+
163+
var back = MarkupTextSerializer.Deserialize(MarkupTextSerializer.Serialize(text, Registry), Registry);
164+
165+
await Assert.That(IsData(back, 1)).IsTrue();
166+
await Assert.That(IsData(back, 0)).IsFalse();
167+
}
168+
}

‎MarkupString/IDataMarkup.cs‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
namespace MarkupString;
2+
3+
/// <summary>
4+
/// A markup that says what the text it covers is, rather than how it looks: that a character is a list
5+
/// separator, say, and not part of an item.
6+
/// </summary>
7+
/// <remarks>
8+
/// <para>No format writes anything for a data layer. The renderer leaves it out before any emitter sees the
9+
/// run, so the text it covers is written as the layers around it would write it, and an emitter registered
10+
/// for a data type is never called.</para>
11+
/// <para>Data stays where visual markup goes: <see cref="MarkupText.WithoutVisualMarkup"/> takes every other
12+
/// layer off and keeps these. Like every layer, it is ignored by <see cref="MarkupText.ToPlainText"/> and
13+
/// equality, carried through slicing, concatenation and joining, and stored through its codec
14+
/// (<see cref="IMarkupCodec"/>); a reader without the codec keeps it as an <see cref="UnknownMarkup"/>.</para>
15+
/// </remarks>
16+
public interface IDataMarkup : IMarkup;

‎MarkupString/MarkupText.cs‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,78 @@ public static MarkupText Join(Func<int, MarkupText> separator, IEnumerable<Marku
154154
/// </summary>
155155
public string ToPlainText() => _plainText ??= WithoutPoints();
156156

157+
/// <summary>
158+
/// The text with every visual layer taken off and its data layers (<see cref="IDataMarkup"/>) kept: what
159+
/// <see cref="ToPlainText"/> gives, plus the data. Points go with their carriers, as they do there.
160+
/// </summary>
161+
public MarkupText WithoutVisualMarkup()
162+
{
163+
if (Runs.IsDefaultOrEmpty) return this;
164+
165+
var hasData = false;
166+
foreach (var run in Runs)
167+
{
168+
foreach (var markup in run.Markups)
169+
{
170+
if (markup is not IDataMarkup) continue;
171+
hasData = true;
172+
break;
173+
}
174+
if (hasData) break;
175+
}
176+
if (!hasData) return Plain(ToPlainText());
177+
178+
var parts = new List<MarkupText>(Runs.Length * 2 + 1);
179+
var position = 0;
180+
foreach (var run in Runs)
181+
{
182+
if (run.Start > position) parts.Add(Plain(Text[position..run.Start]));
183+
position = run.End;
184+
if (IsPoint(run)) continue;
185+
186+
var data = new List<IMarkup>();
187+
foreach (var markup in run.Markups)
188+
if (markup is IDataMarkup) data.Add(markup);
189+
var body = Text.Substring(run.Start, run.Length);
190+
parts.Add(data.Count == 0 ? Plain(body) : Wrap(MarkupSet.Of(data), body));
191+
}
192+
if (position < Text.Length) parts.Add(Plain(Text[position..]));
193+
return Concat(parts);
194+
}
195+
196+
/// <summary>
197+
/// The text with every data layer (<see cref="IDataMarkup"/>) taken off and everything else kept: the
198+
/// counterpart of <see cref="WithoutVisualMarkup"/>, for text whose data must not be trusted.
199+
/// </summary>
200+
public MarkupText WithoutDataMarkup()
201+
{
202+
if (Runs.IsDefaultOrEmpty) return this;
203+
204+
var hasData = false;
205+
foreach (var run in Runs)
206+
{
207+
foreach (var markup in run.Markups)
208+
{
209+
if (markup is not IDataMarkup) continue;
210+
hasData = true;
211+
break;
212+
}
213+
if (hasData) break;
214+
}
215+
if (!hasData) return this;
216+
217+
var builder = ImmutableArray.CreateBuilder<Run>(Runs.Length);
218+
foreach (var run in Runs)
219+
{
220+
var kept = new List<IMarkup>(run.Markups.Count);
221+
foreach (var markup in run.Markups)
222+
if (markup is not IDataMarkup) kept.Add(markup);
223+
if (kept.Count == run.Markups.Count) builder.Add(run);
224+
else if (kept.Count > 0) builder.Add(run with { Markups = MarkupSet.Of(kept) });
225+
}
226+
return new MarkupText(Text, builder.ToImmutable());
227+
}
228+
157229
/// <summary>The same as <see cref="ToPlainText"/>.</summary>
158230
public override string ToString() => ToPlainText();
159231
public bool Equals(MarkupText? other) => other is not null && string.Equals(ToPlainText(), other.ToPlainText(), StringComparison.Ordinal);

‎MarkupString/MarkupTextRenderer.cs‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -379,7 +379,13 @@ private static bool RenderRun(
379379
IBufferWriter<char> output)
380380
{
381381
var body = content.Slice(run.Start, run.Length);
382-
var markups = run.Markups;
382+
// Data is not drawn: no emitter sees it, and text that carries nothing else is written as plain text.
383+
var markups = WithoutData(run.Markups);
384+
if (markups is null)
385+
{
386+
EncodeStretch(content, run.Start, run.End, format.Encoding, output);
387+
return false;
388+
}
383389
var points = CountPoints(markups);
384390

385391
var front = new PooledCharWriter(body.Length + 16);
@@ -499,6 +505,21 @@ private static bool WritePoints(
499505
return written;
500506
}
501507

508+
/// <summary>The run's layers with its data layers taken out, or null when it carried nothing else.</summary>
509+
private static MarkupSet? WithoutData(MarkupSet markups)
510+
{
511+
var data = 0;
512+
foreach (var markup in markups)
513+
if (markup is IDataMarkup) data++;
514+
if (data == 0) return markups;
515+
if (data == markups.Count) return null;
516+
517+
var rest = new List<IMarkup>(markups.Count - data);
518+
foreach (var markup in markups)
519+
if (markup is not IDataMarkup) rest.Add(markup);
520+
return MarkupSet.Of(rest);
521+
}
522+
502523
/// <summary>The run's layers with the points taken out, or null when it carried nothing else.</summary>
503524
private static MarkupSet? Without(MarkupSet markups, int points)
504525
{

‎MarkupString/PublicAPI.Unshipped.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
11
#nullable enable
22
MarkupString.TextEncoding.Markdown = 4 -> MarkupString.TextEncoding
33
static readonly MarkupString.MarkupFormat.Markdown -> MarkupString.MarkupFormat!
4+
MarkupString.IDataMarkup
5+
MarkupString.MarkupText.WithoutVisualMarkup() -> MarkupString.MarkupText!
6+
MarkupString.MarkupText.WithoutDataMarkup() -> MarkupString.MarkupText!

‎docs/custom-markup.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,22 @@ using MarkupString;
1919
public sealed record SpoilerMarkup(string? Reason) : IMarkup;
2020
```
2121

22+
### Data markup
23+
24+
A layer that says what the text *is*, rather than how it looks, implements `IDataMarkup` instead:
25+
that a character separates the items of a list, say, and is not part of one.
26+
27+
```csharp
28+
public sealed record ListSeparator : IDataMarkup;
29+
```
30+
31+
A data layer has a codec and no emitters. The renderer takes it off a run before any emitter sees
32+
the run, so the text is written as the layers around it write it, and an emitter registered for a
33+
data type is never called. `WithoutVisualMarkup()` takes every other layer off and keeps the data
34+
([Transforming](text-operations.md#transforming)). Everything else treats it as any layer: slicing,
35+
concatenation and joining carry it, equality ignores it, and a reader without its codec keeps it as
36+
an `UnknownMarkup`.
37+
2238
## 2. The emitters
2339

2440
Two shapes, and the difference matters:

‎docs/text-operations.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,21 @@ text.Map(part => part.Trim(TrimType.TrimBoth)); // each run and each plain gap o
138138
returns plain text, because the run positions no longer mean anything. `Map` hands you each styled
139139
run and each unmarked gap as its own `MarkupText` and concatenates what you return.
140140

141+
```csharp
142+
text.WithoutVisualMarkup(); // the plain text, with data layers (IDataMarkup) kept
143+
```
144+
145+
`WithoutVisualMarkup` is what stripping colour means when some layers are data: the result is
146+
`ToPlainText()`'s text, points dropped with their carriers, and every `IDataMarkup` layer still on
147+
the characters it covered. Text with no data comes back plain.
148+
149+
```csharp
150+
text.WithoutDataMarkup(); // the same text and colour, with every data layer taken off
151+
```
152+
153+
`WithoutDataMarkup` is the other half: it keeps the visual layers and points and drops the data, for
154+
text from someone who should not be able to say what it is.
155+
141156
## Comparing
142157

143158
```csharp

0 commit comments

Comments
 (0)