Skip to content

Commit 533bff8

Browse files
cknittclaude
andauthored
Fix the formatter's line endings on Windows and in CRLF files (#9042)
* Keep each file's line endings when formatting On Windows, bsc -format wrote its output through text-mode stdout, so every LF became CRLF: rescript format rewrote LF files to CRLF and format --check reported every LF file. Formatting a CRLF file was not stable either, on any platform: the scanner keeps the CRs inside comments, template literals and raw JS, so those lines came out as CR LF beside the printer's plain LF, and text-mode output or an editor turned them into CR CR LF, adding a blank line on each run. Formatting now reads the source with LF line endings and gives the result CRLF ones when the source's first line break is CRLF. bsc -format and res_parser -print res write stdout in binary mode, also inside a captured compiler request, and the editor format command keeps a CRLF document's line endings. With that, Windows CI runs the syntax roundtrip tests and the OCaml and ReScript format checks, whose file lists now go through xargs to stay under the Windows command-line length limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Knittel <ck@cca.io> * Expect formatted files to keep their line endings in rewatch tests The rewatch format unit tests expected formatted output to have CRLF line endings on Windows, which was the text-mode output bug. They now expect each file to keep its own line endings, and check that a formatted CRLF file passes format --check and is left unchanged by format, through the in-process formatter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Knittel <ck@cca.io> * Read syntax test file lists before running tests in the background The syntax test script read each file list in a while loop from a file and started a background job per file. Under MSYS on Windows that occasionally corrupted the names the loop read, so a test file was skipped, or, in the roundtrip tests now running on Windows, reported as a failure. The script now reads each list into an array before it starts any job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christoph Knittel <ck@cca.io> --------- Signed-off-by: Christoph Knittel <ck@cca.io> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent aff4439 commit 533bff8

20 files changed

Lines changed: 320 additions & 82 deletions

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -167,7 +167,7 @@ jobs:
167167

168168
- name: "Syntax: Run tests"
169169
env:
170-
ROUNDTRIP_TEST: ${{ runner.os == 'Windows' && '0' || '1' }}
170+
ROUNDTRIP_TEST: 1
171171
run: ./scripts/test_syntax.sh
172172
shell: bash
173173

‎analysis/src/commands.ml‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -300,6 +300,13 @@ let format ~source ~kind_file =
300300
Lsp.Types.TextEdit.create ~newText:text ~range
301301
in
302302

303+
(* Keep a CRLF document's line endings, rather than return mixed ones *)
304+
let restore_line_endings text =
305+
if Res_line_endings.uses_crlf source then Res_line_endings.to_crlf text
306+
else text
307+
in
308+
let source = Res_line_endings.to_lf source in
309+
303310
let result =
304311
match kind_file with
305312
| Files.Res -> (
@@ -309,16 +316,19 @@ let format ~source ~kind_file =
309316
match List.length diagnostics > 0 with
310317
| true -> Error "Document has syntax errors"
311318
| false ->
312-
Ok (Res_printer.print_implementation ~comments structure |> create_range)
313-
)
319+
Ok
320+
(Res_printer.print_implementation ~comments structure
321+
|> restore_line_endings |> create_range))
314322
| Resi -> (
315323
let {Res_driver.parsetree = signature; comments; diagnostics} =
316324
Res_driver.parsing_engine.parse_interface_from_source ~source
317325
in
318326
match List.length diagnostics > 0 with
319327
| true -> Error "Document has syntax errors"
320328
| false ->
321-
Ok (Res_printer.print_interface ~comments signature |> create_range))
329+
Ok
330+
(Res_printer.print_interface ~comments signature
331+
|> restore_line_endings |> create_range))
322332
| Other -> Error "Failed to format, file not supported"
323333
in
324334

‎changes/format-line-endings.fix.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
- Make the formatter keep each file's line endings. On Windows it no longer rewrites LF files with CRLF or reports them in `rescript format --check`, and formatting a CRLF file no longer adds a carriage return inside multi-line comments, template literals and raw JS on every run.
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
- Run the syntax roundtrip tests and the OCaml and ReScript format checks on Windows CI.

‎compiler/bsc/rescript_compiler_driver.ml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,7 @@ let format_file input =
176176
input
177177
in
178178
match !((Clflags.current ()).output_name) with
179-
| None -> Compiler_request_output.write_stdout formatted
179+
| None -> Compiler_request_output.write_stdout_binary formatted
180180
| Some fname -> Ext_io.write_file fname formatted
181181

182182
let set_color_option option =

‎compiler/ext/compiler_request_output.ml‎

Lines changed: 48 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ type sink = {
55
prefix: string;
66
buffer: Buffer.t;
77
mutable file: (string * out_channel) option;
8+
mutable exact: bool;
9+
(* The buffer holds the final bytes: once binary output is written, text
10+
is converted as it is written instead of when it is read. *)
811
}
912

1013
type streams = {
@@ -55,8 +58,30 @@ let write_sink sink text position length =
5558
| Some (_, channel) ->
5659
output_string channel
5760
(text_mode_input ~win32:Sys.win32 (String.sub text position length))
61+
| None when sink.exact ->
62+
Buffer.add_string sink.buffer
63+
(text_mode_line_endings ~win32:Sys.win32
64+
(String.sub text position length))
5865
| None -> Buffer.add_substring sink.buffer text position length
5966

67+
let write_sink_binary sink text =
68+
match sink.file with
69+
| Some (_, channel) ->
70+
flush channel;
71+
set_binary_mode_out channel true;
72+
output_string channel text;
73+
flush channel;
74+
set_binary_mode_out channel false
75+
| None ->
76+
if not sink.exact then (
77+
let converted =
78+
text_mode_line_endings ~win32:Sys.win32 (Buffer.contents sink.buffer)
79+
in
80+
Buffer.clear sink.buffer;
81+
Buffer.add_string sink.buffer converted;
82+
sink.exact <- true);
83+
Buffer.add_string sink.buffer text
84+
6085
let sink_channel sink =
6186
match sink.file with
6287
| Some (_, channel) -> channel
@@ -65,15 +90,22 @@ let sink_channel sink =
6590
Filename.open_temp_file ~mode:[Open_text] sink.prefix ".log"
6691
in
6792
(* Output written so far goes first. The channel is in text mode, so it
68-
already converts line endings. *)
69-
output_string channel
70-
(text_mode_input ~win32:Sys.win32 (Buffer.contents sink.buffer));
93+
already converts line endings, unless the buffer holds final bytes. *)
94+
if sink.exact then (
95+
set_binary_mode_out channel true;
96+
output_string channel (Buffer.contents sink.buffer);
97+
flush channel;
98+
set_binary_mode_out channel false)
99+
else
100+
output_string channel
101+
(text_mode_input ~win32:Sys.win32 (Buffer.contents sink.buffer));
71102
Buffer.clear sink.buffer;
72103
sink.file <- Some (path, channel);
73104
channel
74105

75106
let sink_contents sink =
76107
match sink.file with
108+
| None when sink.exact -> Buffer.contents sink.buffer
77109
| None ->
78110
text_mode_line_endings ~win32:Sys.win32 (Buffer.contents sink.buffer)
79111
| Some (path, channel) ->
@@ -115,6 +147,16 @@ let write_stdout text =
115147
| Some streams -> write_sink streams.stdout text 0 (String.length text)
116148
| None -> output_string Stdlib.stdout text
117149

150+
let write_stdout_binary text =
151+
match current () with
152+
| Some streams -> write_sink_binary streams.stdout text
153+
| None ->
154+
flush Stdlib.stdout;
155+
set_binary_mode_out Stdlib.stdout true;
156+
output_string Stdlib.stdout text;
157+
flush Stdlib.stdout;
158+
set_binary_mode_out Stdlib.stdout false
159+
118160
let write_stderr text =
119161
match current () with
120162
| Some streams -> write_sink streams.stderr text 0 (String.length text)
@@ -134,7 +176,9 @@ let print_stderr text =
134176
exception Failed of {error: exn; stdout: string; stderr: string}
135177

136178
let with_capture action =
137-
let sink prefix = {prefix; buffer = Buffer.create 256; file = None} in
179+
let sink prefix =
180+
{prefix; buffer = Buffer.create 256; file = None; exact = false}
181+
in
138182
let stdout = sink "rescript-compiler-stdout-" in
139183
let stderr = sink "rescript-compiler-stderr-" in
140184
let streams =

‎compiler/ext/compiler_request_output.mli‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,11 @@ val is_active : unit -> bool
66
val stdout_formatter : unit -> Format.formatter
77
val stderr_formatter : unit -> Format.formatter
88
val write_stdout : string -> unit
9+
10+
val write_stdout_binary : string -> unit
11+
(** Write [text] as is, without the CRLF line endings that a text-mode channel
12+
writes on Windows. *)
13+
914
val write_stderr : string -> unit
1015
val print_stdout : string -> unit
1116
val print_stderr : string -> unit

‎compiler/syntax/cli/res_cli.ml‎

Lines changed: 15 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -61,17 +61,13 @@ end = struct
6161
end
6262

6363
module Cli_arg_processor = struct
64-
type backend = Parser : 'diagnostics Res_driver.parsing_engine -> backend
65-
[@@unboxed]
66-
6764
let process_file ~is_interface ~width ~recover ~target ~jsx_version
6865
~jsx_module ~test_ast_conversion filename =
6966
let len = String.length filename in
7067
let process_interface =
7168
is_interface
7269
|| (len > 0 && (String.get [@doesNotRaise]) filename (len - 1) = 'i')
7370
in
74-
let parsing_engine = Parser Res_driver.parsing_engine in
7571
let print_engine =
7672
match target with
7773
| "ml" -> Res_driver_ml_printer.print_engine
@@ -87,18 +83,23 @@ module Cli_arg_processor = struct
8783
exit 1
8884
in
8985

90-
let (Parser backend) = parsing_engine in
86+
(* Read sources with LF line endings, so a CRLF file prints like an LF one,
87+
also inside comments, template literals and raw JS. *)
88+
let load_source () = Res_line_endings.to_lf (Ext_io.load_file filename) in
9189
(* Color the Format tags (e.g. @{<error>...@}) of printed diagnostics *)
9290
Misc.Color.setup None;
9391

9492
(* Special case for tokens - bypass parsing entirely *)
9593
if target = "tokens" then
9694
print_engine.print_implementation ~width ~filename ~comments:[] []
9795
else if process_interface then
98-
let parse_result = backend.parse_interface ~filename in
96+
let parse_result =
97+
Res_driver.parse_interface_from_source ~display_filename:filename
98+
~source:(load_source ())
99+
in
99100
if parse_result.invalid then (
100-
backend.string_of_diagnostics ~source:parse_result.source
101-
~filename:parse_result.filename parse_result.diagnostics;
101+
Res_diagnostics.print_report parse_result.diagnostics
102+
parse_result.source;
102103
if recover then
103104
print_engine.print_interface ~width ~filename
104105
~comments:parse_result.comments parse_result.parsetree
@@ -120,10 +121,13 @@ module Cli_arg_processor = struct
120121
print_engine.print_interface ~width ~filename
121122
~comments:parse_result.comments parsetree
122123
else
123-
let parse_result = backend.parse_implementation ~filename in
124+
let parse_result =
125+
Res_driver.parse_implementation_from_source ~display_filename:filename
126+
~source:(load_source ())
127+
in
124128
if parse_result.invalid then (
125-
backend.string_of_diagnostics ~source:parse_result.source
126-
~filename:parse_result.filename parse_result.diagnostics;
129+
Res_diagnostics.print_report parse_result.diagnostics
130+
parse_result.source;
127131
if recover then
128132
print_engine.print_implementation ~width ~filename
129133
~comments:parse_result.comments parse_result.parsetree

‎compiler/syntax/src/res_driver.ml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -155,11 +155,11 @@ let print_engine =
155155
{
156156
print_implementation =
157157
(fun ~width ~filename:_ ~comments structure ->
158-
Compiler_request_output.write_stdout
158+
Compiler_request_output.write_stdout_binary
159159
(Res_printer.print_implementation ~width structure ~comments));
160160
print_interface =
161161
(fun ~width ~filename:_ ~comments signature ->
162-
Compiler_request_output.write_stdout
162+
Compiler_request_output.write_stdout_binary
163163
(Res_printer.print_interface ~width signature ~comments));
164164
}
165165

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
let uses_crlf source =
2+
match String.index_opt source '\n' with
3+
| Some index -> index > 0 && source.[index - 1] = '\r'
4+
| None -> false
5+
6+
let to_lf source =
7+
if not (String.contains source '\r') then source
8+
else
9+
let length = String.length source in
10+
let converted = Buffer.create length in
11+
String.iteri
12+
(fun index character ->
13+
if
14+
not
15+
(character = '\r' && index + 1 < length && source.[index + 1] = '\n')
16+
then Buffer.add_char converted character)
17+
source;
18+
Buffer.contents converted
19+
20+
let to_crlf text =
21+
let converted = Buffer.create (String.length text) in
22+
String.iter
23+
(fun character ->
24+
if character = '\n' then Buffer.add_char converted '\r';
25+
Buffer.add_char converted character)
26+
text;
27+
Buffer.contents converted
28+
29+
let preserve source print =
30+
let printed = print (to_lf source) in
31+
if uses_crlf source then to_crlf printed else printed

0 commit comments

Comments
 (0)