concept: prevent mix-up between collections with an origin type parameter

Small, fast programming language where indexes are valid and values can't be shared.

It has an infallible, safe way to refer to items and slices stored in consecutive memory which for example enables representing tree-like data structures without segmented memory or plain indexes (along with the need to handle failure and generations for safety).

Hello, world!

fn Greet
    .name name str .buf buf Buf _origin, char
    : .buf Buf _origin, char .span Span _origin =
    ? .buf buf .span |{Opt Span _origin}no . [string]
    ? Buf-char-opt-span-add-str .. string .new "Hello, " [string]
    ? Buf-char-span-add-str .. string .new name [string]
    Buf-char-span-add-str .. string .new "!\n"

skip to more examples

Install with (requires having rust installed)

cargo install --git https://github.com/lue-bird/sloe sloe

concept: each value must be used used exctly once

Passing a value as an argument? Consumes it. Matching a value? Consumes it. Even variables holding plain numbers for example have to be explicitly duplicated when you need to use them in multiple places.

With this:

values know when they aren't used anymore at compile time. Their memory is always explicitly reclaimed. No need for garbage collection or similar
values can be mutated internally without mutation being detectable
representing things that should only be consumed once, like thread join handles
representing things that should be cleaned up in a specific way, like memory that should be freed from a specific origin
guaranteeing properties like non-overlapping pointed memory regions can enable more optimizations, e.g. through llvm's noalias (though I think currently the languages sloe compiles to don't entirely exploit this fact)

This can feel annoying and clunky. Think e.g. Span-length which takes a span and gives back its size and the given span. Not ony is it clunky, it is also often conceptually less constrained than taking an immutable view (like &Span in rust) because Span-length could behind your back return a changed Span (this can also be an advantage but it usually isn't). If you wanted to track where a value changed, this makes things harder.

The big advantage of this rule is how easy it is to understand and how much simpler and faster it is to statically analyze compared to lifetimes or similar.

Further reading if interested: "linear types", article "must move types", nice short explainer in the austral language docs, "mutable value semantics". Sloe once allowed values to be ignored ("leaked"/forgotten) making them "affine types", like rust owned values. This was changed as it was too easy to for example accidentally forget to handle a value in one query case but not the others. Better be safe and explicit.

concept: consecutive memory, stable index collection `Buf`

A collection which can mark some ranges within itself as vacant without moving existing items around (thus invalidating their indexes). This can be used to "return" memory which has become outdated or useless, for example with Buf-remove, Buf-unset-slot-rid and Buf-unset-span-rid. Note that this functionality is entirely optional and you can just use it for temporary builders etc. which never vacate anything before they are scrapped.

Further reading if interested: This concept is often called slot map, reusing memory. In rust, a prominent example is slab. Comparison of various kinds of similar rust collections. There are even fast general purpose allocators based on this concept, for example zig's SmpAllocator or the rust crate "smmalloc"

concept: collections do not handle their items

Similar to allocators, you cannot access, alter or iterate their contained values directly. Collections are seen as storage into which you can add items, build slices etc. Whenever you do so, you'll get (Unset-)slots and (Unset-)spans that assert your permission to access and alter the referenced items as well as your responsibility to announce their release at some point.

The alternative to this would be to make tiny allocations for every slot and small span and to allow recursive types. This is not uncommon in languages like rust. However, sloe's goal is to do better here and to not bind storage to ownership over its items. Instead, we store a big array buffer of each kind and point into it.

concept: prevent mix-up between collections with an origin type parameter

Every created collection has a unique origin. A value whose type contains an origin can't escape the scope of it's origin. This is checked at compile-time for the expression following origin creation but you'll likely realize it before then:

fn Some-buf . : Buf ??origin cannot even be annotated??, u32 =
    ^buf-origin
    ? Buf-empty{u32} buf-origin [buf]
    ? Buf-add .buf buf .new 123 u32 [.buf buf .slot slot]
    ...
    buf

# compiles
fn Add-some-values buf Buf _origin, u32 : Buf _origin, u32 =
    ? Buf-add .buf buf .new 123 u32 [.buf buf .slot slot]
    ...
    buf

Further reading if interested: The insight "marking origin-specific types specific to code unique paths" has been described similarly in "The Unreasonable Effectiveness of Naming Integers". With the small difference that in sloe's case the unique origin types only exist at compile-time and can thus mark spans, slots, unset spans, unset slots, bufs etc. generically. Additionally it is checked that actually only one collection and its indexes are marked that way.

The idea of "fresh, distinct type instances by code" seems to generally be called "path-dependent types". In rust I know of 2 crates that successfully implement this: https://docs.rs/compact_arena/0.5.0/compact_arena/index.html (safe, pragmatic, simple but bare-bones) and https://docs.rs/indexing/0.4.1/indexing/ (safe, cumbersome, complicated). The same idea but with runtime checking instead of compile-time checking can quite easily be implemented by storing an ID in each collection and the same id in each contained slot, and incrementing a global variable (or similar) for the next available ID: https://github.com/thomcc/handy/blob/master/src/lib.rs#L111-L126 (apart from security this is hardly ever worth it for regular users, considering it is also slower).

I find it interesting that "storage" and "ownership over said storage" are decoupled. I've heard this being called "call-site dependency injection" which also perfectly applies to the idea of passing allocator, interner, concurrency runtime etc. around. I really like this idea but understand that it cannot be implemented in e.g. rust which needs to store its allocator in it's value body to guarantee its content isn't splattered across different inaccessible allocator memories (and to satisfy Drop and to keep most of the existing function interfaces as well as convenience). Sloe solves this dilemma by assigning this unique origin at the high cost of user convenience. In my opinion this isn't quite a solved problem and if you have other ideas, I warmly encourage you to explore and share them.

examples

creating new origins, slots and spans

origin some-name creates a new variable of type origin and a unique local type. Like every other sloe value, an origin type can only be used once, so only for one collection.

# use a temporary collection contained within a scope
fn Use-buf . : u32 =
    ^buf-origin
  	? Buf-empty{u32} buf-origin [buf]
  	? Buf-add .buf buf .new 123 u32 [.buf buf .slot first-slot]
  	? Buf-remove .buf buf .slot first-slot [.buf buf .item first]
  	? Buf-add-array .buf buf .new ; 456 u32 ; 789 u32 [.buf buf .span after-first]
    ...
  	first # = 123 u32

# different branches, different scopes
fn Use-opt opt Opt u32 : ... =
    # this won't compile as their origins come from different branches
    ? (
        ? opt
        [|no .]
            ^buf-origin
            Buf-empty{u32} buf-origin
        [|yes number] (
            ^buf-origin
            ? Buf-one .origin buf-origin .item number [.buf buf .slot slot]
            ...
            buf
            )
        )
    [buf]
    # this will compile:
    ^buf-origin
    ? (
        :opt
        [|no .]
            Buf-empty{u32} buf-origin
        [|yes number] (
            ? Buf-one .origin buf-origin .item number [.buf buf .slot slot]
            ...
            buf
            )
        )
    [buf]
    ...

# tree structure. every slot and span exclusively belongs to that expression.
# If passing so many origins seems annoying to you, check the documentation of Origin
ty Expression _expressions-origin, _patterns-origin, _chars-origin
    |int i32
    |string Opt Span _chars-origin
    |buf Opt Span _expressions-origin
    |call
        .function Slot _expressions-origin
        .arguments Span _expressions-origin
    |lambda
        .parameters Span _patterns-origin
        .result Slot _expressions-origin

ty State _expressions-origin
    # ...patterns, chars, positions etc
    .expressions Buf _expressions-origin, Expression _expressions-origin
    .root-expression Expression _expressions-origin

fn Initial-state
    .expressions-origin expressions-origin Origin _expressions-origin, _expressions_part
    : State (Origin _expressions-origin, _expressions_part) =
    .expressions Buf-empty{Expression _expressions-origin} expressions-origin
    .root-expression (..do parsing..)

fn State-to-interfaces-into
    .interfaces interfaces Buf _interfaces-origin, Interface State _expressions-origin
    .state state State _expressions-origin
    : Buf _interfaces-origin, Interface State _expressions-origin =
    ? (
        Buf-one
        .origin interfaces-origin
        .item |{Interface State _expressions-origin}console-log "hello"
        )
    [.slot slot .buf interfaces]
    ...
    interfaces

pass in origins or collections from the outside

fn Buf-empty{_item} Origin _origin, _part : Buf (Origin _origin, _part), _item

Used by most initializer functions which return new collections from nothing, e.g. for the initial persistent application state. For most other functions, it's more common to pass in an existing collection that you want to edit. (If you're wondering what _part is here: It enables creating an origin inside the function and still passing collections etc using that origin out of the function via Origin-erased. Look it up if you think the existing origin stuff is too restrictive)

syntax

Goal: coherent, practical and compact, avoiding parens and indentation especially for trailing syntax. Sloe is a very explicit language, so any extra verbosity is not tolerable.

# line comment

# number type, so for example
3.2 f32 # number types are p32, u32, i32, f32

# text of type str
"hello"

# unicode scalar of type char
'a'

# most identifiers
variable-or-field-or-variant-or-type-without-parameters-2012

# constructor name
Function-name-or-type-with-parameters

# function call of type Fn in, out. always 1 argument, no parens needed
Some-function argument

# very rarely functions may require type arguments: <...>.
# Which types are needed is shown in the function declaration in <...>s (see later)
Some-function{type}{arguments} Inner-call-as-the-argument inner-call-argument

# record. if values are open-ended they need to be parenthesized.
# The last field value can end in a record without needing to be parenthesized
.first-field first-value .second-field second-value

# empty record, like void/unit.
# commonly used for variants without a value,
# for lazily constructing a value, for empty state/context
# or as the result of functions like U32-rid
.

# ..spread a record into other fields
# Can be placed anywhere and multiple are allowed
.field-1st value-1st .. one-existing-record .. another .field-2nd value-2nd

# temporary array
; first-item ; second-item ; third-item

# local function of type fn.
# the pattern must add a type to all variables
# can **not** use variables from the outer scope.
[parameter-pattern] result

# pattern variable
# appending a type is only necessary and allowed in function parameters
some-variable some-type

# pattern match, checked for exhaustiveness. expressions must be parenthesized if they themselves end in a query.
# The last case result does not need to be parenthesized
? value [first-case-pattern] first-result [second-case-pattern] second-result

# introduce a new origin (describes which collection slots and spans point into).
# The given name can be used as a variable and its unique local type.
# Below will create a variable `new-origin-name` of type `Origin new-origin-name, .`
^new-origin-name  expression-that uses new-origin-name

# introduce multiple new origins with the same unique local type but different part names.
# Below will create
#   - json of type Origin view-origin, .json .
#   - html of type Origin view-origin, .html .
#   - char of type Origin view-origin, .char .
# Not only can this reduce the amount of type variables floating about,
# it's also important for wrapping values into an `Origin-erased`
^ .json .html .char view-origin

# project function declaration.
# For type variables in the result that aren't used in the input,
# functions require appended type parameters: <...>
fn Function-name{_potential}{_type-arguments}{_only-used-in-the-result}
    parameter-pattern-with-types
    : result-type =
    # optional documentation
    # comment
    result-expression

# type name without arguments. lowercase
u32

# type with multiple arguments. Uppercase name.
# Arguments before the last must be parenthesized if they end in a type with arguments
Span origin
My-function-type-alias env, input, output

# declare a shorthand for an existing type
ty point .x i32 .y i32

# can also accept parameters
ty Pair _potential, _type-parameters

# a "choice type" that can come in different shapes ("variants")
# which each have a unique name and one associated value.
|first-option .
|second-option Buf _potential, u32
|third-option Type-name-alias _potential, _type-parameters

# creating a variant value. Note that the type could refer to a type alias
# or a choice type directly <|... ...>
|{a-choice-type}some-variant its value

# variant pattern
|some-variant its value

editor setups

zed/gram-like

clone this repo
open the editor command panel
"zed: install dev extension" or "gram: install extension from folder" and select the cloned-path-sloe/zed directory

Optionally for more precise syntax highlighting, add the setting "semantic_tokens": "combined" or "languages": { "sloe": { "semantic_tokens": "full" } }.

Optionally for a file icon in the project panel, open the editor command panel, select Icon theme selector: toggle and choose "sloe icon dark"/"sloe icon light".

vscode-like

pre-built

download https://github.com/lue-bird/sloe/blob/main/vscode/sloe-0.1.0.vsix
open the command bar at the top and select: >Extensions: Install from VSIX

build from source

clone this repo
open vscode/
run npm run package to create the .vsix
open the command bar at the top and select: >Extensions: Install from VSIX

helix

write to ~/.config/helix/languages.toml:

[language-server.sloe]
command = "sloe lsp"
[[language]]
name = "sloe"
scope = "source.sloe"
injection-regex = "sloe"
file-types = ["sloe"]
indent = { tab-width = 4, unit = "    " }
language-servers = [ "sloe" ]
auto-format = true

For other editors, there's usually a way to specify sloe as the language server and or point to the directory tree-sitter/ in this repository.

As a user of sloe you can stop reading here. The rest is for developers and those interested in language design

known limitations & design weaknesses

What I'm unhappy with in the current design. Writing these down has already helped a lot in coming up with fixes (e.g. Unset-slot, Buf-span-add-own-span, Origin-erased etc. did not exist at one point but were created in response to now deleted list items). And even if I'm unable to fix them, other people/teams might (in other projects)!

it seems quite natural to represent a span of structs as e.g. .field-names Span _field-names .field-values Span _values. This pattern is more memory efficient and can reduce the amount of origins and Bufs necessary. The biggest missing convenience to make this attractive might be helpers to fold over many spans simultaneously. Honestly, the current "fold over one span and step through the rest with Span-start" is annoying. I do not particularly like it as there is always "overspill" that needs to be handled. Additionally, this wastes memory for the duplicated memory (length u32 per extra Span) and wastes computation for unnecessarily handling

Zig "fixes" this by both
- introducing special syntax and crashing at runtime if lengths differ
- only storing the length in one of multiple slices and documenting the expected length for the other start pointers
I think something like introducing Span2 FirstOrigin, SecondOrigin for 2 up to maybe 5 makes sense. You'd be able to fold, access etc. them together and even split those up into separate spans whenever desired (but not join them back!). The sad thing is that this is positional and individual span slots then do not have an associated name. Also, how would this work with existing buf APIs? Something like Buf2-opt-span-add
minor: sometimes, you really own all the items of a buf in one place (especially when the buf items can be trivially copied). Splitting it into opt span+Buf is annoying and wastes a bit of space (length is carried twice and start is always 0)
by default, most passed arguments are quite fat on the stack (e.g. Buf is 6 usize-wide and you may pass a bunch of them). Pointers are much thinner. This can in some parts be optimized by the target language compiler
currently syntax is not full-word-search friendly. Think _type-variable and minus-dash-hyphen
the language is very sequential by design which disqualifies it from running fast on much of parallel computing e.g. GPUs, threads that share memory etc. Sloe is most likely not the right vehicle to explore this space, still it seems like a warning sign for a supposed "general-purpose language"
number types, vector/array types etc. are very underbaked in sloe. I need more real-world experience for their uses. Granted, sloe support for them is only realistic if rust (and zig) improve their support as well

potential improvements in the future

inline Origin paramters (origin, part) into Slot, Span, Buf, Unset-slot Unset-span. This is less confusing IMO but clutters types a little in core signatures
when in query case pattern record, suggest field name in completion
add field and variant rename and references
add code action for spreading a pattern variable
similarly, add "add remaining query cases" code action

add Unset-untracked API to make deconstructing Bufs less reliant on opitimizers figuring out that allocating and tracking vacant spans is useless when all those spans are deallocated anyway at the end.

ty Unset-untracked _origin
    # scattered memory spaces whose locations are not tracked.
    # This is effectively temporarily leaked memory
    # which will only be reclaimed when you call `Buf-with-unset-untracked-rid`.
    # As a result, be very careful when trying to keep a value of this type
    # in persistent memory or simply avoid it.
    # 
    # The usual way to give back slots and spans is to use operations like
    # `Buf-remove`, `Buf-unset-slot-rid`, `Buf-unset-span-rid` etc.
    # However, this internally marks these spaces as vacant,
    # and doing this work may hinder the optimizer in figurinng out
    # that you for example want to scrap the whole Buf (which should be a no-op).
    # ```sloe
    # ^origin
    # ? Buf-empty{u32} origin [buf]
    # ? Buf-add .buf buf .new 20 u32 [.buf buf .slot slot0]
    # ? Buf-add-array .buf buf .new ; 1 u32 ; 2 u32 [.buf buf .span span12]
    # ? Buf-untrack .buf buf .slot slot0 [.buf buf .untracked untracked .item sum]
    # ? (
    #     Span-fold
    #     .span span12
    #     .state (.buf buf .untracked untracked .sum sum)
    #     .step
    #     [
    #     .state (.buf Buf (Origin _origin, .), u32 .untracked Unset-untracked origin .sum sum u32)
    #     .slot Slot origin
    #     ]
    #     ? Buf-untrack .buf buf .slot slot [.item item]
    #     .buf buf
    #     .untracked untracked
    #     .sum U32-add-clamp .a sum .b item
    #     )
    # [.buf buf .untracked untracked .sum sum]
    # Buf-with-unset-untracked-rid .buf buf .untracked untracked
    # ``
    # If you're wondering why this even needs to be a value in the first place:
    # - silently leaking memory of a persistent Buf is fairly nasty. Now you have to store an `Unset-untracked` as a ~mark of shame~ reminder of which Buf may have unreachable memory
    # - `Origin-unerase` requires and checks that `Buf-unerase` and similar can't hit unset memory. Having an `Unset-untracked` makes it so you can't pass this check

fn Unset-untracked-none . : Unset-untracked _origin
fn Unset-untracked-merge
    .a Unset-untracked _origin
    .b Unset-untracked _origin
    : Unset-untracked _origin
fn Buf-untrack
    :
    .buf Buf _origin, _item
    .untracked Unset-untracked _origin
    .item _item
    # short for Buf-unset followed by Buf-unset-slot-untrack
fn Buf-unset-slot-untrack
    .buf Buf _origin, _item
    .span Unset-slot _origin
    :
    .buf Buf _origin, _item
    .untracked Unset-untracked _origin
fn Buf-unset-span-untrack
    .buf Buf _origin, _item
    .span Unset-span _origin
    :
    .buf Buf _origin, _item
    .untracked Unset-untracked _origin
fn Buf-opt-unset-span-untrack
    .buf Buf _origin, _item
    .span Opt Unset-span _origin
    :
    .buf Buf _origin, _item
    .untracked Unset-untracked _origin
fn Buf-vacant-untrack
    Buf _origin, _item
    :
    .buf Buf _origin, _item
    .untracked Unset-untracked _origin
fn Buf-with-unset-untracked-rid
    .buf Buf _origin, _item
    .untracked Unset-untracked _origin
    : .

This still works with Origin-erase due to not being rid-able without ridding the Buf. Open question: there could be an API to recover untracked unset spaces. Is there a use-case? I believe not because otherwise you could have just used

suggest full parameter field patterns of existing project fns (just as rust does). This is super convenient, especially because stuff like expressions Buf _expressions, Expression _expressions _patterns _types doesn't exactly roll easily over one's keyboard
add Set _origin, _item along with add something like Map _origin, _key, _value (or just Map _origin, _item where key is derived from item) which still gives out Slot Origins for each entry but can be queried by key or similar. Map-empty will require providing an .order (Fn .a _key .b _key, .a _key .b _key .order order) .dup (Fn _key, .a _key .b _key) or similar. Alternatively, check if implementing in userland via e.g. index map, AVL or red-black tree backed by a regular Buf is fast enough
consider adding Buf-counting and slot which can reference a slot that is already in use:
```
fn Buf-counting-slot-dup
    .buf Buf-counting _origin, _item
    .slot Slot _origin
    :
    .buf Buf-counting _origin, _item
    .a Slot _origin
    .b Slot _origin
    =
# what about spans?
```
In theory, this would enable graph structures, child-parent relations, doubly-linked lists, inlined string storage (although that would need e.g. Set-counting) etc. Things I dislike with this design:
- access via Buf-counting-unset which does not guarantee seems maybe too difficult (first un-occupy all known slots and even then there is no guarantee). Buf-counting-update should work nicely, especially for copiable types at the cost of: cannot access the buf at the same time and spooky action at a distance
- every item is reference-counted. A slot to a known single-reference item cannot be represented. This is not a biggie because I don't know if there is a use for this
- maybe also -counting versions of map/set etc.
The alternative is of course to do Slot-weak and generational indexes. However, this is un-usable for e.g. inlined string storage and also comes with overhead and even less guarantees.

Open question of representation:
- Buf<{ count: u32/16, item: Item }>: Finding vacant slots takes linear time. Generally fast. Takes the least space on the stack
- { items: Buf<Item>, counts: Buf<u32/16> }: Finding vacant slots takes linear time. Generally fastest. A bit more error-prone than single buf
- { items: Buf<Item>, vacant: Buf<u32>, occoupied_counts: Buf<NonZeroU32/16> }: Finding vacant slots takes constant time but doesn't feel deterministic. Generally fastest but vacating is more expensive. More error-prone than single or double-buf. Takes most space on the stack
- { items: Buf<Item>, counts: Buf<{ range: Range, count: u32/16 }>: Tough to handle and error-prone. Inefficient for cases where slots are handled one by one (no spans exist). Efficient for things like inline storage where spans are clearly defined.
- the above but with vacant ranges and occupied counts split
I think I prefer not storing counts in ranges, as for example for string interning, you could store counted spans in separate collections:
```
chars # of type Buf _chars, char
names # of type Buf-counting _names, Span _chars
..other bufs pointing into chars, e.g. for number literals..
```
This is likely the better option anyway (even though it "hops twice") as it makes searching for the right span possible (and reasonably fast)
introduce ascii (in rust backed by std::ascii::Asci which is currently experimental, in zig backed by u7), require char literals to be suffixed with a type, (optionally provide ascii as a choice type like std::ascii::Char). Change str to chars and ascii to asciis. Preferably rust would support this directly, otherwise do transmutions or similar at some point. Also introduce ascii-to-char, asciis-to-chars and the inverse operations which return opt. remove 'c' syntax in favor of "c" char/ascii

add ascii operations like

fn Char-if-ascii-to-lower char : char
fn Char-if-ascii-to-upper char : char
fn Char-is-ascii-lower char : Opt ascii
fn Char-is-ascii-upper char : Opt ascii
fn Ascii-rid
fn Ascii-dup
fn Ascii-to-u32
fn Ascii-order
fn Ascii-to-lower ascii : ascii
fn Ascii-to-upper ascii : ascii
fn Ascii-is-lower ascii : Opt ascii # maybe |yes.|no. instead
fn Ascii-is-upper ascii : Opt ascii # maybe |yes.|no. instead

change unicode \u{hex} syntax to \u() because {} is used for types
combine scc stuff into the parser state to avoid walking the whole AST for info we could already have collected. Comes at the cost of a thicker ParseState, probably still worth. For extra convenience, it may be reasonable to implement some ByteDecode and ByteEncode traits in rust directly, so that in the common case that the state type is fully known you can hot reload with close to no glue code
add byte-level APIs, like Buf-opt-span-take-i32 enianness and Buf-opt-span-take-f32 enianness. Ultimately, these sould allow got reloading or simple byte protocols in general
(probably not that good of an idea) to the above effect, it could be nicer to add ultra-basic macro support, so e.g. !u32 "3" where u32 is of type _fn str, |success u32 |failure str (instead of 3 u32) which would evaluate the given function (which should return |error str (?) |ok Value). This would allow userland to create e.g. hex parsing functions, arabic number systems, string raw bytes stuff etc.
(probably not that good of an idea) consider not counting function calls as using up a function variable. The disadvantage is that "overplacing" a variable step-wise doesn't work anymore if not wrapped somehow. Maybe a small price to pay And where no Length field can be instantiated
consider replacing kebab-case with camelCase/PascalCase. while I do much prefer the typing experience of kebab-case, camelCase is shorter (!!), think BufCharOptSpanAddStr compared to Buf-char-opt-span-add-str (5 chars less, 20%!) and potentially more readable (?) due to clearer distinction to _ and (this won't matter as much if call and construct syntax does not involve _). Take a bigger example, convert the case and see how it feels
(not fully sure) Add explicit field punning syntax: Add pattern syntax _ (untyped) / _ value-type (typed) (and maybe expression syntax _) where _ behaves like a variable with the name of the parent. So e.g. .field (_ value-type) would introduce a variable named field. and pattern |variant _ would introduce a variable named variant. Likewise, linked-list-cons .nodes _ .linked-list numbers .new 3 u32 would work if a variable named nodes exists. If no parent name exists, an error is thrown. The only goal here is making record patterns more convenient to work with (similar to swifts named parameters). The biggest worry I have is name clashes. Things like rename might also become a little more complex. I would normally not consider this as a feature, but since sloe is so painfully explicit, I feel users deserve some sugar for their effort.
add field spread syntax for types where overlapping field names is okay as long as their value types are equal
add variant spread syntax ||existing-choice-type |other-variants-before-and-or-after (only in types) analogue to the field spread syntax
when checking, avoid shortcutting early when possible, still traversing sub-items even when a clear error has been found
add c# compilation as well if there is demand
verify that origin creation is correct for all kinds of recursion! e.g. this one seems on the edge of correct: different bufs have the same origin but their slots can't intermix.
```
fn Recurse
    .consume-origin consume-origin Origin _consume-origin, .
    .result-origin result-origin _result-origin
    : Buf _result-origin, u32 =
    ^local-origin
    ? Buf-empty{u32} consume-origin [temporary]
    ? Recurse local-origin result-origin [result]
    ? Buf-add .buf temporary .new 1 u32 [.slot slot .buf temporary]
    ...
    result
```
If we find a problem, creating a new origin should be disallowed in (mutually) recursive calls. This is a bit restrictive but alright I believe. If feeling motived, look into proof languages and make sure this is rock solid
improve memory efficiency of string operations (currently buf of char). This is probably inefficient because:
- more work on program boundaries. E.g. instead of validating data, then reusing the bytes, we need to re-allocate them and then finally un-convert them into utf-8 anyway
- most bytes are 3/4th 0s because ascii is so common. wasted space is bad for the cache and memory usage If these somehow turn out to be nonconcerns (e.g. through array-of-union(enum) optimizations) that would be cool as well since Buf _, char is a much nicer API to work with
(not possible with origin-erased probably) zig-only: store an allocator within an origin (but! what about unset_slice? That one should probably store an allocator, too, and re-allocate if the buf origin allocator reference differs. I think this can be slightly unintuitive for sloe users but should in practice be okay). This achieves that origins created from within sloe code are arena-allocated and origins from user code are (usually) not, choosing e.g. MemoryPool.Aligned (does that actually work even?)
switch to a symmetric unerase API which enforces that all values inside are unerased. This would allow Buf-origin-erased to be removed in favor of Buf (Origin erased, _part) and Origin-erased-rid/Origin-erased-map to just work. To that end, introduce
```
fn Origin-unerase
    .erased Origin-erased .origin Origin _o
    : Origin-isolated _o
fn Origin-isolated-split
    Origin-isolated _o, .a _a .b _b
    : .a Origin-isolated _o, _a .b origin-isolated _o, _b
```
but I couldn't find something reasonable for choice types
(once there is an easy way to check if a pointer is aligned in rust) change cast_or_rid_and_allocate to recover alignment differences if the address happens to align
(once allocator API is stabilized) allocate all collections with an origin that was declared in sloe using a locally-passed impl Allocator<>
I think in theory there should be all the bits and pieces present to allow for struct-of-arrays and arrays-of-variant-values (made up name). E.g. internally compiling
- Buf Origin, .a A .b B to A·B<Buf<A>, Buf<B>>
- for Buf Origin, |a A |b B to either
  - Tag·ValueIndex·A·B<Buf<A_or_B_Tag>, Buf<u32>, Buf<A>, Buf<B>> (which also has ~2 hops but makes sense when sizes of A and B are different enough)
  - A·B<Buf<A>, Buf<B>> (which requires the index to hold both the tag and the value index, aka 64 bit instead of 32, which somewhat defeats the point of reducing padding of the variant when values get bigger. Potentially there could however be struct-of-arrays for individual variant values making this worth it: https://github.com/dist1ll/osmium & https://alic.dev/blog/dense-enums)
  - A·B<Buf<A>, Map<u32, B>> (which is inefficient, and wasteful if B is common, and also doesn't scale with more than 2 variants)
look into soa_derive for rust, maybe this already does most of the useful work
(very out of scope but thinking never hurts) imagine what a logic programming language with this concept would look like. I imagine it wouldn't look much different (!) though with some different tradeoffs (e.g. more complex stdlib and compiler output, potentially a different typing and exhaustivess system)

rejected ideas

As a hobby language that deliberately cannot by itself interface with the operating system, C etc. we can afford to skip many complex features. First some smaller-scale rejected ideas

allow expressions whose type is known (basically anything except inputs to queries) to omit extra type info (namely number, |{}variant and project-fn{}). I'm a little torn because this makes construction inconsistent and increases the distance between the known type and expression. On the other hand this is already the case for query case patterns (deliberately so) but has a much higher convenience gain there
add special syntax fn-once that automatically assembles the environment from the used local variables. Rejected in favor of more explicit construction with contextual names and potentially multiple fns. More info in "not coherently formulated thoughts"
add tuples: (* a * b * c). I dislike them conceptually but operations like U32-add or U32-dup are nicer with them. The field names .a .b are just noise. These can also be used for the Array type paramerer. Adding tuples might make for a nicer user interface when calling from rust
consider allowing origin name at the project scope. This allows reducing the number of type parameters flying around in things like Expression _expressions, _patterns, _types, _source, _cases, ... if desired. It also makes initial_state much easier to call from the rust side (though we need to be careful how...). Rejected because this makes it more or less impossible to run multiple sloe instances from a single rust program Issue is that in general single-return-continuation is rare in sloe
requiring all (!) generic type parameters to be passed to calls I feel like this is more "natural", easier to type-check but way more verbose / redundant. And especially because having many origin type variables is common, this sadly won't fly
adding function call syntax sugar similar to piping. While this is bloody wonderful (succinct, intuitive-ish, great for builders), it doesn't quite have much of a purpose which pattern matching doesn't fill well already. But more importantly it is quite limiting (requires positional arguments, requires them in the right order, doesn't apply to variants and similar). It also introduces "yet another way of writing the same code" which is dislike
(rejected, but interesting in theory) making Buf etc store multiple kinds of data (heterogenous) and letting them give out Slot origin, data-type and Span origin, data-type. This means that usually only one origin needs to be passed to things like expression and slots/spans actually tell you what data they point to. Similarly, only one buf needs to be passed around. This makes the porpose of Buf being allocator-ish spaces rather that collections to query and edit more clear and makes passing them around to operations very simple, e.g. expression-end .expression Expression _origin .data Buf _origin, ... : .buf Buf _origin ... .end text-position. This would also in theory enable a crazy representation of tagged unions as:
```
ty Expression-slot _origin
    |int Slot _origin, i32
    |plus Slot _origin, .left Expression-slot _origin .right Expression-slot _origin
    ...
```
This also means slices etc need to be stored separately in the origin buf. The issue currently is that it feels hard to optimally construct/query such a heterogenous structure. Its structure must be created at compile-time. Dynamically this doesn't fly: Buf origin = { bucket: Map<for type_byte_size: { key: type_byte_size, value: Buf<type_byte_size> }> }. However, really providing this in sloe would require sloe to add some kind of "type variable must be record" constraint:
```
^buf-origin
? Buf-empty{.expression expression .pattern Pattern buf-origin} buf-origin [buf]
? Buf-add .buf buf .new some-expression [.buf buf slot some-expression-slot]
? Buf-add .buf buf .new some-pattern  [.buf buf .slot some-pattern-slot]
...
```
This is probably doable in zig but hardly in rust without significant macro magic. Any ideas welcome!
allowing .. (|variant ...) with a single variant and untyped variant expressions. No, should consistently use single-field record
field and variants are changed so field names and variant names are uppercase and . is spread (same for |), e.g.
```
ty event
    |Counter-clicked
    |Mouse-moved .X u32 .Y u32
```
The benefit is that the question above is answered (single field = single variant). Overall this is "more correct" than the current solution. Rejected because this is harder to type (and would require a change of type variable syntax)
(rejection not final for all eternity. If you have a good use case, I'll support it) allow field and variant names to start with digit, upper-case and -, like fn Char-dup char char : .0 char .1 char. One nice thing is that this matches what most language use as field names for tuples. This is also a little bit confusing but you don't have to use it. Use cases are e.g. ty bit |0 . |1 ., type board-pin |0 . |1 . |3 . |10 . and nicer array records. Not included currently for consistency and simplicity.
switch from error{OutOfMemory}! to anyerror! for ease of use with external functions. Rejected because zig errors should be explicitly handled by sloe

why no `&mut`/`inout`

While seemingly convenient and magnitudes better than regular mutable pointers,

it's less obvious than passing values through
there's no easy way to change the name of a resulting value that represents something different now
there's no way to "reconstruct" a different out value. Especially for non-trivial edits the &mut approach can get messy or it's straight up impossible and parts will need to get cloned unnecessarily
there's no way to change the type (e.g. from Opt Span to Span)
there's no there's two ways to specify most conversions, with usually no clear method of converting one to the other
it's surprisingly common that one path consumes an argument, the other path keeps it in tact (e.g. when searching a tree with intermediate information. Either we find something, consuming the context or we come up empty-handed with the original context, like fn .context context ... : |exit found |go-on context where found contains some parts of the context). This isn't modelled well with &mut
&mut means the resulting changed collection is not returned, making use as the input to another function impossible. This almost necessarily results in the classic procedural-style statement form as opposed to the functional-style expression form. Minor gripe: especially in languages that don't allow local scopes with local returns (far, far too many) this basically makes it impossible to locally introduce a value, change it and implant it somewhere; instead you have to move the variable up to the top level.
returning . (like returning Unit in gleam) feels super awkward to my brain. Most often, languages then automatically return void/... in the absence of a return and introduce all kinds of constructs like re-assignable variables, additional constructs for looping and branching that all can only return void/... . To my brain, this just confuses matters; it loves simple to follow flow of state!
&mut usually comes with the need to check for non-overlapping references to the same parts of data. This isn't possible with owned data passing in the first place
&mut usually necessitates the need for offering the same APIs in two shapes, e.g. make_uppercase(&mut self) vs to_uppercase(self)->Self on rust's char/str types, take()/take_mut(), std::mem::swap etc. This to me just feels wrong
it's hard to be precise in what parts of a type are captured exclusively, shared or unused

rusts immutable references & have some similar trade-offs but seem kind of unavoidable at least for languages like rust.

why no closures that capture environment variables automatically like in rust

its type cannot be specified. as such, it cannot generally be stored as part of a type
no clear unified interface. There could be multiple functions, there could be an output that additionally returns the captured values (allowing it to be called again), there could be an output that only sometimes returns the captured variables, etc.
I personally never had a need for this. Usually you can just make the environment a type variable and you're golden

I'm strangely really convinced that this is the obvious, correct design decision (for most programming languages at that!). Note that the current design does not natively have a dyn Fn; it needs to be manually emulated via an explicit | choice type.

why no traits / type classes / (duck) (static) dispatch

traits introduce a crazy amount of complexity
If really necessary, traits can be represented using arguments. I have yet to hit any complexities with this.
attaching a set of functions to one "subject" seems super strange to me. Operations usually take different objects and create something new
traits create a "one-fits-all" interface. Thinking to e.g. rusts clone(&), drop(&mut) or to_string(&), they seem super sensible in concept but fall short when trying to clone into a different allocator, trying to use a different string representation, or trying to mutate some backing storage on drop. These restrictions can sometimes kind of be circumvented in annoying ways. It all just seems so arbitrary for no apparent reason.
traits are usually touted as a sensible solution for operator overloading. sloe does not have operators
traits push languages in the direction of nominal record and choice types. This isn't wrong per se but to me they usually start to feel clunky to use and I then use them less often then would be helpful (e.g. for long parameter lists or multiple outputs)

Because traits cover a vast theoretical area of use, they tend to be used a bunch. I've never found them particularily pleasant to use. Libraries often only expose some functionality through these, without proper documentation. Incidentally, I've also found editor tooling to be lacking in these areas, not knowing if you want to look at the general or specific function.

why no (mathematical) operators

operators introduce a good amount of complexity: infix (and prefix) notation, associativity, precedence, most likely a way to overload based on context
edge-case behavior (e.g. saturating vs overflowing vs checked vs carry vs ...) should be easier to control
in general, operators are concise but as a result quite ambiguous. For example, changing a boolean to an integer may silently not generate a compiler error when ! is binary not, or when changing a list to a string with ++
while numbers, bool and bit operations are not that uncommon, there are features that would deserve these symbols more, even in typical imparative languages (think return, switch { case }, structure, import, public, static, void, null, ...)
allowing infix - and prefix - leads can lead to very confusing situations like call-1 but more importantly using - as an operator pretty much prevents languages from using the superior (easier to type) kebab-style for identifiers

Somehow despite it's issues (math syntax kind of sucks, even the tiny subset), operators are one of the most prevalent features in programming languages, even hobby and experimental ones (0th class citizen). I do not quite understand this (well I guess not adding operators adds to the weirdness budget) .

why no single-field access

a.k.a record.field. Quick and easy answer: Because this makes it embarassingly easy to forget handling a field (now or in the future). I've identified this as the second most common source of bugs in my own code. And in sloe, not handling a field could mean leaking some memory, so it would be even worse potentially!

why no positional function arguments

with positional arguments it isn't really possible to make the last argument open ended (at least with keeping the current syntax)
it's tough (usually) to annotate a function whose arguments and argument types are unknown. E.g. what would fn-dup's type be?

positional arguments (usually) means no passing in bulk

fn U32-square-clamp natural u32 : u32 =
    U32-add-clamp U32-dup natural

"Positionality" in general is pretty much absent in sloe. E.g. positional arguments are super convenient, so they tend to be used for everything, even arguments that would benefit from a clear description. Sloe had positional arguments once, largely because the rust-sloe interface is simpler in rust with positional arguments.

unnecessary features in sloe

Features I've added which are fully replacible by other existing features. If you're looking to learn from sloe, maybe do not learn from these:

record spread. It provides an alternative syntax sugar for something that could already be expressed. I originally introduced it to make builders like string builers less jarring but I'm not so sure this worked. I'm on the fence; if you have complaints I'll remove this feature
nested pattern matching. It's existence makes compilation, exhaustiveness-checking, error messages and flow-typing-like matching (e.g. matching |a in <|a|b|c> leaving |b|c) harder. It also creates a "two modes of matching" problem: You e.g. can't match on numbers, chars, strings, span start and lengths etc. And so you sometimes need an extra step, leading to nested matches anyway (does not feel consistent). It also "takes control from the user int othe magic hands of the compiler" and thus it may run checks etc. in a different order than you have. I originally introduced it to make e.g. matching on multiple Opts easier. It helps keep context clear and visible like "if the left sub is empty and the right sub is a branch with an empty left side, do this". Honestly I should not have been so hasty to add this feature
stack-allocated array syntax. It provides an alternative syntax sugar for something that could already be expressed as repeated queried function calls. I'm convinced that a feature like this would be very asked for if it didn't exist. Adding a bulk of items to a buf seems very useful on first sight because
- all kinds of examples and tests start with manually adding items. Doing this one by one seems like cringe busywork.
- building uis or any kind of trees programmatically, you more than often end up needing to specify sub-nodes of a parent. Adding this bulk of nodes as an array is only natural and avoids so much noise.
Well, what are the alternatives, then?
- simply provide Buf-opt-span-add2/3/4/5/6/7/8 etc. While it really doesn't feel good, it's not very far from solutions of production languages, see e.g. java's list.of2/3/4/.... (Though adding or removing items means adjusting the number which is annoying, especially for the argument field names)
- provide and suggest better primitives and helpers. For example, instead of providing a list of modifiers, it may just make sense to e.g. provide a record of options or use builder-style helpers for the individual properties
- introduce syntactical diabetis or macro-esque bullshit for repeated function calls
```
...
_@0 stack-cons .. example-stack .new 39 u32
& _@0 ..@ .new 3 u32
& _@0 ..@ .new 6 u32
& _@0 ..@ .new 9 u32
```
  (the above is obviously insane in a bad way, but there may be a middle-ground)
So yeah, these aren't amazing either.

general quetions you might have

does sloe fill any niche well enough to be worth it?

I'd say domains where languages like safe rust or performance-aware C#/swift/go stand today:

not extensive enough to have a front seat in systems programming, but comfortably sitting on top of a somewhat thin platform layer
not as easy to use as scripting languages like python, gleam, lua, elm, prolog, etc.
mainly used for tools, applications or similar where maintainability, robustness and being easy to reason about is important

Don't be afraid to program in a language sloe compiles to for tasks sloe feels annoying to use for. E.g. I imagine writing a recursive file watcher in sloe is not fun, so just "outsource" it :)

why put work into transpiling to existing languages

The best user experience interfacing with sloe code from existing system-level languages is directly generating code in that language. Just sharing type names, structs, tagged unions, function signatures etc without any work by you is tasty enough. And if you end up outgrowing sloe, you have all the code right there (that's the hope anyway but output readability is likely wose than as if it was hand-written). Being easy to transpile is an explicit goal of sloe, enabled by its very limited set of features.

why write the compiler and tooling in rust?

It did that before and it does it's job. I imagine the current style leaves some performance on the table but I'd be surprised if it was too slow for its only potential user, the human reading this (<3).

dev setup

to re-compile

cargo install --offline --debug --path . sloe

TODO

do introduce query pattern record spread syntax after all. It enables the "use the defaults except" pattern which would be inpossible annoying otherwise:
```
Some-fn
? Some-fn-defaults [.. all .except except]
? Except-rid except [.]
.. all .except new-value
```
I've changed my mind on this being okay because you need to handle all fields anyway
track down formatting bug which can duplicate the last declaration (maybe related: document ends in unrecognized code). Then change error message of type construct with missing argument to explaining that types with no arguments are lowercase
add Buf-(opt-)span-update which asks for .span Span _origin .item-update Fn _item, _item. Same for Opt Span. This functionality is already possible but unnecessarily inconvenient
add Buf-span-fold and Buf-opt-span-fold. Their functionality is already covered but inconvenient considering how common that operation is
add Buf-opt-span-add-repeat, Buf-span-add-repeat, Buf-opt-span-add-repeat-for-length-positive, maybe even unfold
(not fully sure) add Buf-opt-unset-span-add-length-positive, Buf-opt-unset-span-add-length, Buf-unset-span-add-length, Buf-unset-span-add-own-opt-span
try to recover typed pattern without a variable more nicely by when all other cases fail trying to parse a type and representing it as a variable without a variable
give nicer error when only a field is missing or too much
find some way to generate nicer IDE type displays. Maybe tabs work?
remove Origin-erased-rid. It can't really be made useful
try to make accidentally used _ in identifiers more gentle
"No local variable in scope has this name." should list available variable names

consider adding

fn Origin-erased-map
    .erased Origin-erased _value-erased
    .change Fn _value-erased, _value-erased-new
    :
    Origin-erased _value-erased-new

check Buf lengths after every append in rust the same way as done in zig and js but panic instead
fix comment TODOs

not coherently formulated thoughts

on collections not owning items

In rust, collections tend to own their item data, so safely keeping references reaching inside is tough. Alternatively, we could reach for Range<usize> and usize but we've lost ties to the origin structure and rust does not (yet?) have a mechanism for temporarily assuming actual ownership over some part of a parent structure. This relationship is flipped on it's head in sloe: All items of collections are divided into slots and spans which are owned by the code that parked values there in the first place.

Honestly this idea seems "obviously" useful and it's surprising I can't find other languages that lean into it (there is rust which at least enables it in userland). I assume one reason is that linear types are required in some part to avoid leaks all over the place.

One way this helps is that nested collections aren't segmented: what is usually Buf<Box<str>> aka n separate memory pieces can be e.g. Buf ... Span str-origin + Str str-origin (in rust there are I think crates like oroborus for this)

on shadowing

since each variable can be used at most once, most introduced names that would traditionally be considered "shadowed" are aready out of scope in sloe. When their scopes actually overlap though, you'll get an error

on defer

I love how linear types somewhat mirror the functionality of defer ...getRidOfIt(); but without the yucky control flow. All operations happen in the specified order in sloe! This also simplified code generation

sorting?

sorted-span etc. could be nice (only in userland most likely!) That in combination with binary/interpolation search could be a nice alternative to set and map collections. Needs Buf-span-sort (issue: how to implement in rust and js?). Not really possible with that approach: Buf-span-sorted-insert. It takes O(n) time.

Also take another look at index maps. It's probably a good idea anyway to add hashing helpers to sloe. Would be nice if index maps could in some way make use of Bufs, for example if it's like

ty Map _slots, _in-hash-order, _item
    .slots Buf _slots, Slot _in-hash-order
    .items Buf _in-hash-order, _item

Name		Name	Last commit message	Last commit date
compile		compile
example-hello-world-js		example-hello-world-js
example-hello-world-rust		example-hello-world-rust
example-hello-world-zig		example-hello-world-zig
example-linked-list		example-linked-list
example-website-rust		example-website-rust
social		social
tree-sitter		tree-sitter
vscode		vscode
website		website
zed		zed
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
UNLICENSE		UNLICENSE
main.rs		main.rs
readme.md		readme.md

Latest commit

History

Folders and files

Repository files navigation

concept: each value must be used used exctly once

concept: consecutive memory, stable index collection Buf

concept: collections do not handle their items

concept: prevent mix-up between collections with an origin type parameter

examples

creating new origins, slots and spans

pass in origins or collections from the outside

syntax

editor setups

zed/gram-like

vscode-like

pre-built

build from source

helix

known limitations & design weaknesses

potential improvements in the future

rejected ideas

why no &mut/inout

why no closures that capture environment variables automatically like in rust

why no traits / type classes / (duck) (static) dispatch

why no (mathematical) operators

why no single-field access

why no positional function arguments

unnecessary features in sloe

general quetions you might have

does sloe fill any niche well enough to be worth it?

why put work into transpiling to existing languages

why write the compiler and tooling in rust?

dev setup

TODO

not coherently formulated thoughts

on collections not owning items

on shadowing

on defer

sorting?

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

concept: consecutive memory, stable index collection `Buf`

why no `&mut`/`inout`