A distributed background job processing system built in Go, with persistent job storage, concurrent workers, and asynchronous task execution.
Server receives HTTP request from client to perform a job and validates it. A valid job request is stored in the database with a PENDING status. Application developers create workers (eventually using a library) that spin up, connect to Spool and advertise the job type that they can execute and how many of those jobs they can execute concurrently. Spool schedules all of the work in a fault-tolerant and fault-resistant way.
When the server is run, it starts by applying all database migrations. Then, it instantiates the Services and Repositories for the system, passes them to the ConnectRPC Service Handlers generated by Buf and serves the RPCs on a Go HTTP Server. The procedures then interact with the database using the jackc/pgx library and return the response to the client.
The client is a collection of generated Connect-Go Clients which are instantiated at startup time as Go HTTP Clients.
The communication protocol and between the client and the server is defined using protobuf with the Buf CLI tool to generate the structs. Run buf lint to check the current .proto files agains the linter and buf generate to compile them and generate the Go code they describe. Use just buf-breaking to check for breaking changes between the current changes and the previous ones. The buf workspace is defined in buf.yaml and the specifications for generating the code is in buf.gen.yaml.
The application uses the ConnectRPC protocol for communication between the client and the server. This is done using the Connect-Go library which allows to define type-safe Remote Procedure Calls using our protobuf definition. We use the protovalidate library to ensure the invariants are respected on all the fields passed between client and server.
This project uses a PostgreSQL database inside of a docker container. The database can be spun up using docker compose up and lives on port 5432. Once the database is running, you can connect to it by using the just db-connect command.
Migrations to the database are applied using the golang-migrate library. The server always tries to migrate up to the latest schema on startup. Migrations can also be applied or reversed manually using the CLI tool by running just db-migrate <up|down> [amount]. Leaving amount blank will apply or reverse all migrations. To create a new migration, run just db-create <name>. See documentation for more details.
The data models live in this folder. Each file contains the error types, model to Protobuf representation mappers and the models themselves.
The repositories translate database errors into domain errors and return them to the services. The services then add business information to the errors before converting them to connect.Errors and returning them to the client.
Centralised system configuration is found in the config folder. This provides a single place where you can change properties of the system.
Current properties are:
- Server port on localhost
- just : Open-source command runner
- golang-migrate : Open-source database schema migration tool
- Buf CLI : Open-source protobuf toolchain
- Docker Compose : Define and run multi-container applications
- Connect-Go : Open-source library for connectRPC
- Lefthook : Fast and simple pre-commit and pre-push manager
- Testing
- Make the error handling and mapping for the job repo and service more robust