AGENTS.md

AGENTS.md

Instructions for AI coding agents working in this repository.

Project Overview

Go RESTful API starter kit built with Echo, GORM, JWT, Redis, MongoDB, 2FA, email verification, and password recovery. Module path: github.com/pilinux/gorest. Requires Go 1.26.0+ (go.mod declares go 1.26.0; CI tests against Go 1.26.x and 1.27.x).

Build and Run Commands

Build

go build -v ./...

Tidy Dependencies

go mod tidy

Format

go fmt ./...

Vet

go vet -v ./...

Lint

golangci-lint run ./...
revive ./...

CI uses golangci-lint v2.13.2 with --timeout 5m --verbose.

Test - All

go test -v -cover ./...

Test - Single Package

go test -v -cover ./lib/...
go test -v -cover ./config/...

Test - Single Test Function

go test -v -run TestHashPass ./lib/...
go test -v -run TestFirewall ./lib/middleware/...

Test Environment

Tests require environment variables. In CI these come from secrets. Locally, source setTestEnv.sh before running tests:

source setTestEnv.sh

example3/ is the exception: its tests use in-memory fakes for both Mongo interfaces, so go test ./example3/... needs neither env vars nor a live database.

Cross-Platform Vet (CI runs all six)

GOOS=linux GOARCH=amd64 go vet -v ./...
GOOS=darwin GOARCH=arm64 go vet -v ./...

Project Structure

• config/ - Configuration loading from .env files
• controller/ - HTTP request handlers (thin layer, calls handler)
• handler/ - Business logic layer (called by controller)
• service/ - Utility services (auth, security, email)
• database/ - DB connection management and models
• database/model/ - Shared data models (Auth, HTTPResponse)
• database/migrate/ - GORM auto-migration
• lib/ - Core utility library (hashing, encryption, validation)
• lib/middleware/ - Echo middleware (JWT, CORS, firewall, rate limit)
• lib/renderer/ - HTTP response rendering (JSON and HTML)
• lib/server/ - Graceful server shutdown
• example/ - Legacy example app
• example2/ - Recommended example app (interface-driven, DI)
• example3/ - Envelope-encryption example app (MongoDB-only, interface-driven)

example2/ Structure (Recommended)

example2/
├── cmd/app/main.go          # Entry point
└── internal/
    ├── database/
    │   ├── migrate/migrate.go  # Auto-migration
    │   └── model/models.go     # App-specific GORM models
    ├── handler/                # HTTP handlers (Echo context binding)
    ├── repo/                   # Repository pattern (data access)
    ├── router/router.go        # Route definitions + middleware setup
    └── service/                # Business logic

example3/ Structure (Envelope Encryption)

Same layered shape as example2/, MongoDB-only, and the one example with a full unit-test suite (fakes for both Mongo interfaces, so it runs with no live database).

example3/
├── cmd/app/main.go          # Entry point: config → mongo → indexes → key load → serve
└── internal/
    ├── database/model/         # Mongo documents + request/response DTOs
    ├── handler/                # Thin Echo handlers (+ tests)
    ├── repo/                   # Mongo persistence; index.go bootstraps indexes
    ├── router/router.go        # Route definitions + middleware setup
    └── service/                # KeyManager, text/number crypto, file crypto
        ├── scheme.go           # Pinned HKDF labels, chunk size, envelope.Scheme
        └── storage.go          # File-id validation, size limit, the sized and sizeless seals

Key points when working in example3/:

• Crypto primitives come from github.com/pilinux/crypt/envelope; example3 only pins its own HKDF labels in service/scheme.go. Changing those labels orphans every already-stored item.
• A master key is generated once and stored wrapped by a KEK derived from ENCRYPTION_SECRET; ENCRYPTION_SECRET_OLD triggers a re-wrap at boot.
• repo.EnsureIndexes must run before the master key is loaded: the unique index on keyName is what keeps concurrent boots from storing two master keys.
• Files use the streaming + padded API (SealPaddedStreamAAD / SealPaddedAtAAD / OpenPaddedReaderAAD), with the file id as AAD. The padded format hashes its own domain tag in beside that AAD, so the two openers can never be swapped: reading a padded file back with OpenReaderAAD fails with ErrStreamAuth, and the caller passes the bare file id either way.
• Padding needs the payload length before chunk 0 carries it, but every upload is one pass. A raw application/octet-stream body declares its length in Content-Length, so sealSized → SealPaddedStreamAAD seals it in order. A multipart/form-data part declares nothing (browsers send no per-part Content-Length), so sealSizeless → SealPaddedAtAAD (new in crypt v0.0.29) counts it while it streams: chunks 1 onward are sealed at the offsets the format fixes for them and chunk 0, which holds the frame, is sealed last and written at offset 37. That needs a seekable destination, which is why sealTo hands the open *os.File to the sealer rather than an io.Writer. Nothing is ever staged: no plaintext is written to disk, and the cost of a sizeless seal is one extra chunk of memory, not a second pass over the bytes.
• POST files/encrypt/unpadded seals the same stream format without padding (SealStreamAAD), which needs no length at all and writes strictly in order; the file size then reveals the plaintext length. Both routes store into the same collection and directory, and one decrypt/delete pair serves both: model.FileRecord.Unpadded (omitempty, so zero = padded) is what tells OpenForDownload which opener and framing to use. Flipping that flag cannot reinterpret a file; the padded format tag makes it fail to authenticate.
• internal/handler/ is the thin Echo layer (the controller role in the table below, despite the name): it binds the request, calls the service and renders. The (model.HTTPResponse, int) contract lives one layer down, in internal/service/, whose methods take a context.Context first and use named returns.
• Handler tests are external (package handler_test); service tests are internal (package service) because they reach unexported helpers.

Layer Responsibilities

Layer	Package	Responsibility
Controller	controller/	Bind request, call handler, render response
Handler	handler/	Business logic, validation, returns (HTTPResponse, int)
Service	service/	Shared utilities (auth, email, crypto, JWT blacklist)
Repository	example2/internal/repo/, example3/internal/repo/	Data access abstraction (interface-driven)
Database	database/	Connection management (RDBMS, Redis, MongoDB)
Config	config/	Load .env, expose GetConfig(), feature checks

Controller-Handler 1:1 Mapping

Every controller function in controller/ is a thin wrapper that binds the HTTP request and calls the corresponding function in handler/. Controllers use renderer.Render() to send the response. There are 19 exported functions in each package with a 1:1 correspondence. Every handler takes a context.Context as its first parameter; controllers pass c.Request().Context() so handlers honor client cancellation/deadlines:

Controller	Handler	Description
CreateUserAuth	CreateUserAuth(ctx, auth model.Auth)	User registration
Login	Login(ctx, payload model.AuthPayload)	User login
Refresh	Refresh(ctx, claims middleware.MyCustomClaims)	Refresh JWT
Logout	Logout(ctx, jtiAccess, jtiRefresh string, expAccess, expRefresh int64)	Invalidate tokens
PasswordForgot	PasswordForgot(ctx, authPayload model.AuthPayload)	Send recovery email
PasswordRecover	PasswordRecover(ctx, authPayload model.AuthPayload)	Reset password
PasswordUpdate	PasswordUpdate(ctx, claims, authPayload)	Change password
Setup2FA	Setup2FA(ctx, claims, authPayload)	Generate 2FA secret
Activate2FA	Activate2FA(ctx, claims, authPayload)	Enable 2FA
Validate2FA	Validate2FA(ctx, claims, authPayload)	Verify OTP
Deactivate2FA	Deactivate2FA(ctx, claims, authPayload)	Disable 2FA
CreateBackup2FA	CreateBackup2FA(ctx, claims, authPayload)	Generate backup codes
ValidateBackup2FA	ValidateBackup2FA(ctx, claims, authPayload)	Use backup code
VerifyEmail	VerifyEmail(ctx, payload model.AuthPayload)	Verify email
CreateVerificationEmail	CreateVerificationEmail(ctx, payload model.AuthPayload)	Resend verification
VerifyUpdatedEmail	VerifyUpdatedEmail(ctx, payload model.AuthPayload)	Verify email change
UpdateEmail	UpdateEmail(ctx, claims, req model.TempEmail)	Submit new email
GetUnverifiedEmail	GetUnverifiedEmail(ctx, claims)	Get pending email
ResendVerificationCodeToModifyActiveEmail	ResendVerificationCodeToModifyActiveEmail(ctx, claims)	Resend email change code

All handler functions return (httpResponse model.HTTPResponse, httpStatusCode int).

Import Alias Conventions

Always use these aliases when importing gorest packages:

import (
    gconfig "github.com/pilinux/gorest/config"
    gcontroller "github.com/pilinux/gorest/controller"
    gdb "github.com/pilinux/gorest/database"
    gmodel "github.com/pilinux/gorest/database/model"
    ghandler "github.com/pilinux/gorest/handler"
    glib "github.com/pilinux/gorest/lib"
    gmiddleware "github.com/pilinux/gorest/lib/middleware"
    grenderer "github.com/pilinux/gorest/lib/renderer"
    gserver "github.com/pilinux/gorest/lib/server"
    gservice "github.com/pilinux/gorest/service"
)

Also alias logrus: log "github.com/sirupsen/logrus"

Code Style Guidelines

Imports

Organize imports in three groups separated by blank lines:

1. Standard library
2. External packages (use aliases where conventional)
3. Internal packages (github.com/pilinux/gorest/...)

import (
    "errors"
    "net/http"
    "strings"

    "github.com/labstack/echo/v4"
    log "github.com/sirupsen/logrus"

    "github.com/pilinux/gorest/config"
    "github.com/pilinux/gorest/database/model"
)

Naming Conventions

• Exported functions/types: PascalCase (e.g., CreateUserAuth)
• Unexported functions/vars: camelCase (e.g., authFinal)
• Acronyms stay uppercase: JWT, CORS, RDBMS, HTTP, 2FA
• Packages: lowercase, single word (e.g., middleware, renderer)
• Test structs: camelCase (e.g., hashPassTest, hashVerifyTest)
• Error variables: prefix with err (e.g., errThis, errHash)

Struct Tags

Use json:"fieldName,omitempty" for JSON tags and gorm:"primaryKey" style for GORM tags. Hide sensitive fields with json:"-". Mark non-DB fields with gorm:"-".

type MyCustomClaims struct {
    AuthID  uint64 `json:"authID,omitempty"`
    Email   string `json:"email,omitempty"`
    Role    string `json:"role,omitempty"`
}

Function Signatures

Handlers take a context.Context first and return (model.HTTPResponse, int) where the int is the HTTP status code. Use named return values with bare returns for early exits:

func CreateUserAuth(
    ctx context.Context,
    auth model.Auth,
) (httpResponse model.HTTPResponse, httpStatusCode int) {
    // ...
    httpResponse.Message = "wrong email address"
    httpStatusCode = http.StatusBadRequest
    return
}

Error Handling

• Log errors with log.WithError(err).Error("error code: XXXX.X")
• Use numbered error codes (e.g., 1002.1, 1002.2) for tracing
• Return user-facing messages via httpResponse.Message
• Never expose internal errors to API consumers
• Use errors.New() for simple errors, fmt.Errorf with %w for wrapping

if err != nil {
    log.WithError(err).Error("error code: 1002.1")
    httpResponse.Message = "internal server error"
    httpStatusCode = http.StatusInternalServerError
    return
}

HTTP Responses

Always use model.HTTPResponse for API responses. Use renderer.Render() to send responses in controllers, which handles both JSON and HTML template rendering, and return its error:

return renderer.Render(c, data, http.StatusOK)
return renderer.Render(c, data, http.StatusBadRequest)

Comments

• Package doc comments on every package: // Package name - description
• Function doc comments: // FuncName - description
• Copyright headers where applicable: // The MIT License (MIT) // Copyright (c) 20XX pilinux
• Use numbered step comments for complex logic
• Punctuation is plain ASCII: never an em dash (U+2014). Pick the mark that fits the clause relationship (comma, semicolon, colon, parentheses) or start a new sentence. Applies to comments, docs, and commit messages alike.

Testing

• Use external test packages (package lib_test, not package lib); where a test genuinely needs unexported identifiers, add an export_test.go in the internal package (see lib/export_test.go). example3/internal/service is the one place that tests from inside the package instead.
• Table-driven tests with named struct types
• Use t.Run() for subtests with descriptive names
• Use t.Errorf for assertions (no external assertion library)
• Test helper functions should be unexported

func TestHashPass(t *testing.T) {
    tests := []hashPassTest{
        {name: "blank password", ...},
        {name: "with password", ...},
    }
    // safe pattern: capture per-iteration value for subtests to avoid closure-capture bugs
    for i := range tests {
        tc := tests[i]
        t.Run(tc.name, func(t *testing.T) {
            // ...
        })
    }
}

Core Models and Types

Key Models (database/model)

// Auth - users table (has custom JSON marshal/unmarshal)
type Auth struct {
    AuthID    uint64         `gorm:"primaryKey" json:"authID,omitempty"`
    Email     string         `gorm:"index" json:"email"`
    Password  string         `json:"password"`
    // EmailCipher, EmailNonce, EmailHash: hidden from JSON (json:"-")
    // VerifyEmail: int8 status field (json:"-")
}

// AuthPayload - registration/login request body
type AuthPayload struct {
    Email            string `json:"email,omitempty"`
    Password         string `json:"password,omitempty"`
    VerificationCode string `json:"verificationCode,omitempty"`
    OTP              string `json:"otp,omitempty"`
    SecretCode       string `json:"secretCode,omitempty"`
    RecoveryKey      string `json:"recoveryKey,omitempty"`
    PassNew          string `json:"passNew,omitempty"`
    PassRepeat       string `json:"passRepeat,omitempty"`
}

// HTTPResponse - standard API response wrapper
type HTTPResponse struct {
    Message any `json:"message,omitempty"`
}

// JWTPayload - token pair response
type JWTPayload struct {
    AccessJWT   string `json:"accessJWT,omitempty"`
    RefreshJWT  string `json:"refreshJWT,omitempty"`
    TwoAuth     string `json:"twoFA,omitempty"`
    RecoveryKey string `json:"recoveryKey,omitempty"`
}

// TempEmail - holds data during email replacement
type TempEmail struct {
    Email    string `gorm:"index" json:"emailNew"`
    Password string `gorm:"-" json:"password,omitempty"`
    // + ID, timestamps, cipher fields, IDAuth
}

Key Constants (database/model)

// Email verification statuses
const EmailNotVerified int8 = -1
const EmailVerifyNotRequired int8 = 0
const EmailVerified int8 = 1

// Email type constants (for SendEmail)
const EmailTypeVerifyEmailNewAcc int = 1
const EmailTypePassRecovery int = 2
const EmailTypeVerifyUpdatedEmail int = 3

// Redis key prefixes
const EmailVerificationKeyPrefix string = "gorest-email-verification-"
const EmailUpdateKeyPrefix string = "gorest-email-update-"
const PasswordRecoveryKeyPrefix string = "gorest-pass-recover-"
const Backup2FALimitKeyPrefix string = "gorest-2fa-backup-limit-"

// Redis hash field names: password recovery (stored as a hash) and the
// 2FA backup-code brute-force limiter (also stored as a hash).
const PasswordRecoveryFieldEmail string = "email"
const PasswordRecoveryFieldAttempts string = "attempts"
const Backup2FAFieldAttempts string = "attempts"
const Backup2FAFieldCooldownUntil string = "cooldown_until"

Key Constants (config)

const Activated string = "yes"
const PrefixJtiBlacklist string = "gorest-blacklist-jti:"

Key Constants (database)

// Use errors.Is(err, gorm.ErrRecordNotFound) instead of string matching.
var RedisConnTTL int // context deadline in seconds for Redis connections

// Sentinel errors for guarding against an uninitialized client. The getters
// (GetDB/GetRedis/GetMongo) return nil when the subsystem is disabled.
var ErrDBNotInitialized error    // RDBMS connection is nil
var ErrRedisNotInitialized error // Redis client is nil
var ErrMongoNotInitialized error // MongoDB client is nil

Configuration System

Loading

gconfig.Env()       // loads .env file via godotenv.Load()
gconfig.Config()    // loads all config from env vars into Configuration struct
gconfig.GetConfig() // returns *Configuration

Feature Check Functions

These return bool and are used to conditionally enable features:

gconfig.IsProd()       gconfig.IsRDBMS()      gconfig.IsRedis()
gconfig.IsMongo()      gconfig.IsJWT()        gconfig.Is2FA()
gconfig.IsCORS()       gconfig.IsWAF()        gconfig.IsRateLimit()
gconfig.IsSentry()     gconfig.IsCipher()      gconfig.IsHashPass()
gconfig.IsBasicAuth()  gconfig.IsAuthCookie()  gconfig.InvalidateJWT()
gconfig.IsOriginCheck()          gconfig.IsTemplatingEngine()
gconfig.IsEmailService()         gconfig.IsEmailVerificationService()
gconfig.IsPassRecoveryService()  gconfig.Is2FADoubleHash()
gconfig.IsEmailVerificationCodeUUIDv4()
gconfig.IsPasswordRecoverCodeUUIDv4()

Database Initialization Patterns

RDBMS

// Production pattern uses retry loop
for {
    if err := gdb.InitDB().Error; err != nil {
        fmt.Println(err)
        time.Sleep(10 * time.Second)
        continue
    }
    break
}
db := gdb.GetDB()        // get *gorm.DB connection
gdb.InitTLSMySQL()       // TLS setup for MySQL

Redis

for {
    if _, err := gdb.InitRedis(); err != nil {
        fmt.Println(err)
        time.Sleep(10 * time.Second)
        continue
    }
    break
}
client := gdb.GetRedis() // get radix.Client

MongoDB

for {
    if _, err := gdb.InitMongo(); err != nil {
        fmt.Println(err)
        time.Sleep(10 * time.Second)
        continue
    }
    break
}
client := gdb.GetMongo() // get *mongo.Client

// Index management
gdb.MongoCreateIndex("db", "collection", index)
gdb.MongoCreateIndexes("db", "collection", indexes)
gdb.MongoDropIndex("db", "collection", []string{"field"})
gdb.MongoDropAllIndexes("db", "collection")

Closing

gdb.CloseSQL()     // close RDBMS
gdb.CloseRedis()   // close Redis
gdb.CloseMongo()   // close MongoDB
gdb.CloseAllDB()   // close all (returns error)

Middleware Patterns

JWT

router.Use(gmiddleware.JWT())           // validate access token
router.Use(gmiddleware.JWT("__session")) // custom cookie name
router.Use(gmiddleware.RefreshJWT())    // validate refresh token

CORS

router.Use(gmiddleware.CORS(cfg.Security.CORS))
gmiddleware.GetCORS()    // returns CORSConfig struct
gmiddleware.ResetCORS()  // reset configuration

Firewall

router.Use(gmiddleware.Firewall("whitelist", "192.168.1.0/24, 10.0.0.1"))
router.Use(gmiddleware.Firewall("blacklist", "192.168.100.0/24"))
router.Use(gmiddleware.Firewall("whitelist", "*")) // allow all
gmiddleware.ResetFirewallState() // reset (useful for testing)

2FA

// Requires JWT middleware first
router.Use(gmiddleware.TwoFA("on", "off", "verified"))

Rate Limiting

limiter, _ := glib.InitRateLimiter("100-M", "X-Real-Ip") // 100/minute
router.Use(gmiddleware.RateLimit(limiter))

Origin Check

router.Use(gmiddleware.CheckOrigin([]string{"https://example.com"}))

Sentry

gmiddleware.InitSentry(dsn, env, version, enableTracing, sampleRate)
gmiddleware.NewSentryHook(dsn, env, ...) // for goroutine-specific loggers
gmiddleware.DestroySentry()
router.Use(gmiddleware.SentryCapture())

Template Rendering

router.Use(gmiddleware.Pongo2("templates/"))

Service Layer Quick Reference

// Auth
gservice.GetClaims(c)                    // extract JWT claims from Echo context
gservice.GetUserByEmail(email, bool)     // look up user
gservice.GetEmailByAuthID(id)            // get email by auth ID
gservice.IsAuthIDValid(id)               // check existence in DB
gservice.ValidateAuthID(id)              // checks zero + DB existence
gservice.ValidateUserID(id, email)       // checks authID != 0 and email != ""

// JWT blacklist
gservice.JWTBlacklistChecker()           // middleware
gservice.IsTokenAllowed(ctx, jti)        // check if token is blacklisted

// Email
gservice.SendEmail(ctx, email, emailType, opts...)  // send via Postmark
gservice.Postmark(params)               // low-level Postmark delivery

// Crypto
gservice.DecryptEmail(nonce, ciphertext) // decrypt stored email
gservice.CalcHash(data, secret)          // BLAKE2b hash
gservice.GetHash(dataIn []byte)         // hash for 2FA key encryption
gservice.RandomByte(n)                  // secure random bytes
gservice.GenerateCode(n)               // random alphanumeric code

// 2FA
gservice.Validate2FA(secret, issuer, otp)
gservice.DelMem2FA(authID)

// 2FA backup-code brute-force limiter (Redis-backed, escalating cooldown;
// no-op when Redis is disabled)
gservice.Backup2FACooldown(authID)         // remaining cooldown (time.Duration, error)
gservice.RegisterBackup2FAFailure(authID)  // record a failure, returns cooldown (time.Duration, error)
gservice.ClearBackup2FALimit(authID)       // reset after success (error)

Utility Library Quick Reference

// Encryption (AES-GCM)
glib.Encrypt(plaintext, key)
glib.Decrypt(ciphertext, key)

// Password hashing (Argon2id)
glib.HashPass(config, password, pepper)

// Key derivation
glib.GetArgon2Key(password, salt, keyLen)

// TOTP / QR
glib.NewTOTP(account, issuer, hash, digits)
glib.NewQR(otpBytes, issuer)
glib.ValidateTOTP(otpBytes, issuer, otp)
glib.ByteToPNG(qrPNG, path)

// Validation
glib.ValidateEmailFormat(email)         // structure/length only, no DNS (deterministic)
glib.ValidateEmail(email)               // structure + live MX lookup (bounded by glib.EmailMXLookupTimeout)
glib.ValidatePath(path, allowedDir)     // prevent traversal
glib.FileExist(path)

// Random
glib.SecureRandomNumber(digits)

// String helpers
glib.RemoveAllSpace(s)
glib.StrArrHTMLModel(s)                // split "k:v;k:v" into []string
glib.HTMLModel(strArr)                  // []string to map[string]any

// Rate limiter
glib.InitRateLimiter(rate, trustedPlatform)

Graceful Shutdown

// Full signature:
// GracefulShutdown(srv *http.Server, timeout time.Duration, done chan struct{}, closeDB ...func() error) error
done := make(chan struct{})
go func() {
    err := gserver.GracefulShutdown(srv, 30*time.Second, done, gdb.CloseAllDB)
    if err != nil {
        fmt.Println(err)
    }
}()

HTTP Response Rendering

// JSON response
return grenderer.Render(c, data, http.StatusOK)

// HTML template response (requires Pongo2 middleware)
return grenderer.Render(c, data, http.StatusOK, "template.html")

Files to Never Read or Modify

Per .gitignore, avoid these paths:

• *.log, .env, *.bak
• vendor/, keys/, tmp/, .vscode/, .build/
• coverage.txt
• pri*key*.pem, pub*key*.pem
• crosscompile*.sh

CI Pipeline

GitHub Actions run on push and PR (path-filtered to **/*.go, go.mod, go.sum, **/*.yml), with a Go version matrix of 1.26.x and 1.27.x:

• go vet on 6 platform combinations
• gosec security scanner
• govulncheck for known vulnerabilities
• Build on 6 platforms (linux/darwin/windows x amd64/arm64)
• Tests with -race and coverage (only on push to main), uploaded to Codecov
• golangci-lint v2.13.2 (separate workflow, --timeout 5m --verbose)

Contributing

• PRs target the main branch
• One commit per unique task
• Test and validate code before committing
• Document new features in README.md