Start dependents once a module's interface is final

Why:
    On builds limited by a long dependency chain most job slots sit idle
    on the critical path: a 136 module library gets only 1.6x from 7
    domains.

When a slot would otherwise idle, a module can now start as soon as each
unfinished dependency's interface is final: an implementation without
.resi has computed the .cmi it is about to write, or an interface file
has compiled. The early request reads those .cmi files from memory and
waits at a gate before writing a file, running a command or returning
its result. The scheduler adopts it at the point it would have started
it (dependencies complete, module still dirty), so its code generation
sees the dependencies' .cmj files, and artifacts, diagnostics and
compiler.log are unchanged. Otherwise it is abandoned without effects,
together with anything that started on top of it.

Redirected paths are matched as the build file store keys them. The
directories in which the build publishes interfaces are spelled from
canonical package roots, both in the redirected paths and on the load
path, so a lookup there that matches no redirected path names another
file. A lookup of a redirected file name in any other directory, which
may be an alias from user flags (with -nostdlib those come first), or
one that differs only in case, waits for adoption instead. Whether an
announced interface changes the published one is read from disk, so
that an abandoned request never fills the store. Modules with
js-post-build commands or lowercase .cmi names don't release their
dependents early, and with one domain nothing changes.

Review: Can a dependent see an interface that differs from master's?
It reads the exact contents the dependency then writes, and it is only
adopted after the dependency completed. If the dependency fails later,
for example in code generation, the dependent is abandoned before it
wrote, ran or reported anything.

Signed-off-by: Jono Prest <jjprest@gmail.com>

Author: JonoPrest

changes/pipeline-dependents-after-interface.polish.md

@@ -0,0 +1 @@
+- Build: start type checking a module as soon as the interfaces of its dependencies are final, instead of waiting for their JavaScript generation, which shortens builds limited by long dependency chains.

rewatch-ocaml/README.md

@@ -78,6 +78,21 @@ going to the process's stdout and stderr. The compiler build identity is
 derived from the compiler sources, so compiler changes clean the build while
 rewatch-only changes don't.
 
+When no ready module can take a free job slot, a module can start before its
+dependencies have finished: as soon as each unfinished dependency's interface
+is final. That is when the implementation of a module without an interface
+file has computed the `.cmi` it is about to write, or when an interface file
+has compiled. The early request reads those not yet published `.cmi` files
+from memory. It keeps its job slot while it waits, before it writes any file,
+runs any command or reports any result, until the scheduler would have
+started it anyway: its dependencies have completed and it still needs
+compiling. Otherwise it is abandoned without effects, as are early requests
+that depend on it. Code generation therefore always sees the `.cmj` of
+completed dependencies, and artifacts, diagnostics and `compiler.log` are
+those of a build without pipelining. A module with `js-post-build` commands
+does not release its dependents early. `REWATCH_SPECULATION_LOG` names a file
+to which tests append each early start, adoption and abandonment.
+
 External PPXs, `js-post-build` hooks, and `--after-build` remain subprocesses
 under the process owners described below.
 

rewatch-ocaml/compile_assets.ml

@@ -128,6 +128,11 @@ let create directories =
         state_entries);
   state
 
+let directories state =
+  Hashtbl.fold
+    (fun directory _ directories -> directory :: directories)
+    state.files_by_directory []
+
 let files state directory =
   Hashtbl.find_opt state.files_by_directory directory
   |> Option.value ~default:[]

rewatch-ocaml/compile_assets.mli

@@ -8,6 +8,10 @@ type t
 
 val is_managed_basename : string -> bool
 val create : string list -> t
+
+val directories : t -> string list
+(** The directories indexed, in which the build publishes interfaces. *)
+
 val files : t -> string -> string list
 val ast_sources : t -> string -> ast_source list
 val ast_dependencies : t -> string -> string list

rewatch-ocaml/compiler_process.ml

@@ -49,8 +49,54 @@ let exit_code = function
   | Unix.WEXITED code -> code
   | Unix.WSIGNALED signal | Unix.WSTOPPED signal -> 128 + signal
 
-let run_in_process ?poll (request : request) =
-  log_compiler_request request;
+(* Paths are matched as the build file store keys them. The directories in
+   which the build publishes interfaces are spelled from canonical package
+   roots, both in a redirected path and on the load path, so a lookup there
+   that matches no redirected path names another file. Any other directory,
+   such as one from user flags, may be an alias of one of them, and a file
+   that differs only in case may be the same one: a lookup there of a
+   redirected file name waits, as the file may be stale until then. *)
+let cmi_redirect ~cwd ~cmi_directories ~wait_for_adoption redirects =
+  match redirects with
+  | [] -> fun _ -> None
+  | redirects -> (
+    let key path =
+      Build_file_store.key
+        (if Filename.is_relative path then Filename.concat cwd path else path)
+    in
+    let table length = Hashtbl.create length in
+    let contents_by_key = table (List.length redirects) in
+    let folded_keys = table (List.length redirects) in
+    let folded_names = table (List.length redirects) in
+    List.iter
+      (fun (published, contents) ->
+        let key = key published in
+        Hashtbl.replace contents_by_key key contents;
+        Hashtbl.replace folded_keys (String.lowercase_ascii key) ();
+        Hashtbl.replace folded_names
+          (String.lowercase_ascii (Filename.basename key))
+          ())
+      redirects;
+    let directories = table (List.length cmi_directories) in
+    List.iter
+      (fun directory -> Hashtbl.replace directories (key directory) ())
+      cmi_directories;
+    fun path ->
+      let key = key path in
+      match Hashtbl.find_opt contents_by_key key with
+      | Some _ as contents -> contents
+      | None ->
+        let folded = String.lowercase_ascii key in
+        if
+          Hashtbl.mem folded_names (Filename.basename folded)
+          && (Hashtbl.mem folded_keys folded
+             || not (Hashtbl.mem directories (Filename.dirname key)))
+        then wait_for_adoption ();
+        None)
+
+let run_in_process ?poll ?(log = true)
+    ?(hooks = Compiler_request_state.no_hooks) (request : request) =
+  if log then log_compiler_request request;
   match List.rev request.args with
   | [] ->
     {
@@ -60,7 +106,7 @@ let run_in_process ?poll (request : request) =
     }
   | input :: reversed_argv -> (
     match
-      Rescript_compiler_driver.run_request ~cwd:request.cwd
+      Rescript_compiler_driver.run_request_with_hooks ~hooks ~cwd:request.cwd
         ~argv:(List.rev reversed_argv) ~input
         ~run_external:
           (Some
@@ -95,15 +141,77 @@ let run ?poll request =
   Option.iter (fun poll -> poll ()) poll;
   result
 
-let task request =
+let task ?(hooks = Compiler_scheduler.no_compile_hooks) request =
   let cancelled = Atomic.make false in
+  let gate = hooks.Compiler_scheduler.gate in
+  (* A speculation logs its call once adopted, on its worker as before, so
+     that the test-only REWATCH_COMPILER_CALL_LOG lists only adopted calls. *)
+  (* A logging failure fails every later gate pass too, the final one outside
+     the compiler included, so nothing is written after it and the task fails
+     as when a request logs at its start. The task runs on one domain. *)
+  let logged = ref (if Option.is_none gate then Ok () else Error None) in
+  let wait_for_adoption () =
+    Option.iter Speculation_gate.wait gate;
+    match !logged with
+    | Ok () -> ()
+    | Error (Some error) -> raise error
+    | Error None -> (
+      match log_compiler_request request with
+      | () -> logged := Ok ()
+      | exception error ->
+        logged := Error (Some error);
+        raise error)
+  in
+  let redirect =
+    cmi_redirect ~cwd:request.cwd ~cmi_directories:hooks.cmi_directories
+      ~wait_for_adoption hooks.cmi_redirects
+  in
+  let compiler_hooks =
+    Compiler_request_state.
+      {
+        before_effect = wait_for_adoption;
+        interface_computed =
+          (match hooks.on_interface_computed with
+          | None -> fun _ _ -> ()
+          | Some computed ->
+            fun path bytes ->
+              computed
+                (if Filename.is_relative path then
+                   Filename.concat request.cwd path
+                 else path)
+                bytes);
+        redirect_cmi =
+          (match gate with
+          | None -> redirect
+          | Some gate ->
+            fun path ->
+              (* Interfaces load throughout type checking: an abandoned
+                 speculation stops at the next one. *)
+              if Speculation_gate.is_abandoned gate then
+                raise Speculation_gate.Abandoned;
+              redirect path);
+      }
+  in
   Process.concurrent_task
-    ~cancel:(fun () -> Atomic.set cancelled true)
+    ~cancel:(fun () ->
+      Atomic.set cancelled true;
+      Option.iter Speculation_gate.cancel gate)
     (fun () ->
-      run_in_process
-        ~poll:(fun () ->
-          if Atomic.get cancelled then raise (Process.Interrupted 15))
-        request)
+      (* An abandoned speculation that has not started yet does no work. *)
+      Option.iter
+        (fun gate ->
+          if Speculation_gate.is_abandoned gate then
+            raise Speculation_gate.Abandoned)
+        gate;
+      let result =
+        run_in_process ~log:(Option.is_none gate) ~hooks:compiler_hooks
+          ~poll:(fun () ->
+            if Atomic.get cancelled then raise (Process.Interrupted 15))
+          request
+      in
+      (* A speculation's result, and its publication, wait for adoption. *)
+      wait_for_adoption ();
+      result)
 
 let run_tasks ?poll ?on_complete tasks =
   Process.run_tasks

rewatch-ocaml/compiler_process.mli

@@ -27,7 +27,22 @@ val run_parses :
     successful parse that printed nothing with {!publish_parse} on the worker
     that parsed it. The caller publishes the parses that printed warnings. *)
 
-val task : request -> Process.task
+val task : ?hooks:Compiler_scheduler.compile_hooks -> request -> Process.task
+
+val cmi_redirect :
+  cwd:string ->
+  cmi_directories:string list ->
+  wait_for_adoption:(unit -> unit) ->
+  (string * string) list ->
+  string ->
+  string option
+(** [cmi_redirect ~cwd ~cmi_directories ~wait_for_adoption redirects] maps a
+    resolved .cmi candidate path to the contents redirected for it, matching
+    paths as {!Build_file_store} does. [cmi_directories] are the directories in
+    which the build publishes interfaces. A candidate that may name a
+    redirected file under another spelling, because it differs from one only
+    in case or lies in another directory under the same name, calls
+    [wait_for_adoption] and is then read as usual. *)
 
 val namespace_task :
   runtime:string ->