Skip to content

bug: Grain eager relocation ignores WithActivationRole #1334

Description

@leonlau

Bug description

A Grain created with WithActivationRole("game-worker") is initially activated only on a
node advertising that role. After its owner leaves, eager relocation may reactivate the
Grain on a node that does not advertise game-worker.

This differs from the expected role-constrained placement behavior and makes it possible
for gateway/coordinator-only nodes to unexpectedly host worker Grains.

How to reproduce it?

The following self-contained sample uses one gateway node and two game-worker nodes.
Start the gateway first so it becomes the oldest node/relocation leader. The sample needs
a NATS server but no application-specific services.

Project layout:

grain-role-repro/
└── main.go

main.go:

package main

import (
	"context"
	"flag"
	"fmt"
	"log"
	"os"
	"os/signal"
	"strings"
	"syscall"
	"time"

	"github.com/tochemey/goakt/v4/actor"
	discoverynats "github.com/tochemey/goakt/v4/discovery/nats"
	"github.com/tochemey/goakt/v4/remote"
)

type markerActor struct{}

func (*markerActor) PreStart(*actor.Context) error  { return nil }
func (*markerActor) Receive(*actor.ReceiveContext)  {}
func (*markerActor) PostStop(*actor.Context) error  { return nil }

type roleGrain struct{}

func (*roleGrain) OnActivate(_ context.Context, props *actor.GrainProps) error {
	log.Printf("ACTIVATED grain=%s node=%s:%d",
		props.Identity().Name(), props.ActorSystem().Host(), props.ActorSystem().Port())
	return nil
}
func (*roleGrain) OnReceive(ctx *actor.GrainContext) { ctx.NoErr() }
func (*roleGrain) OnDeactivate(_ context.Context, props *actor.GrainProps) error {
	log.Printf("DEACTIVATED grain=%s node=%s:%d",
		props.Identity().Name(), props.ActorSystem().Host(), props.ActorSystem().Port())
	return nil
}

func main() {
	id := flag.String("id", "gateway", "node label")
	rolesCSV := flag.String("roles", "gateway", "comma-separated cluster roles")
	discoveryPort := flag.Int("discovery-port", 33220, "discovery port")
	peersPort := flag.Int("peers-port", 33221, "cluster peers port")
	remotingPort := flag.Int("remoting-port", 33222, "remoting port")
	activate := flag.Bool("activate", false, "activate the test Grain after the cluster settles")
	flag.Parse()

	roles := strings.Split(*rolesCSV, ",")
	discovery := discoverynats.NewDiscovery(&discoverynats.Config{
		NatsServer:    "nats://127.0.0.1:4222",
		NatsSubject:   "goakt.grain-role-repro.v1",
		Host:          "127.0.0.1",
		DiscoveryPort: *discoveryPort,
	})
	cluster := actor.NewClusterConfig().
		WithKinds(&markerActor{}).
		WithRoles(roles...).
		WithPartitionCount(7).
		WithReplicaCount(2).
		WithPeersPort(*peersPort).
		WithDiscoveryPort(*discoveryPort).
		WithMinimumPeersQuorum(1).
		WithBootstrapTimeout(15 * time.Second).
		WithClusterStateSyncInterval(300 * time.Millisecond).
		WithDiscovery(discovery)

	system, err := actor.NewActorSystem("grain-role-repro",
		actor.WithRemote(remote.NewConfig("127.0.0.1", *remotingPort)),
		actor.WithCluster(cluster),
		actor.WithShutdownTimeout(30*time.Second),
	)
	if err != nil {
		log.Fatal(err)
	}
	ctx := context.Background()
	if err := system.Start(ctx); err != nil {
		log.Fatal(err)
	}
	if err := system.RegisterGrainKind(ctx, &roleGrain{}); err != nil {
		log.Fatal(err)
	}
	log.Printf("STARTED id=%s roles=%v remoting=%d", *id, roles, *remotingPort)

	if *activate {
		time.Sleep(8 * time.Second)
		identity, err := actor.GrainOf[*roleGrain](ctx, system, "role-bound-grain",
			actor.WithActivationRole("game-worker"),
			actor.WithActivationStrategy(actor.RoundRobinActivation),
			actor.WithGrainEagerRelocation(),
		)
		if err != nil {
			log.Fatal(err)
		}
		log.Printf("CREATED identity=%s", identity.String())
	}

	stop := make(chan os.Signal, 1)
	signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM)
	<-stop
	stopCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()
	if err := system.Stop(stopCtx); err != nil {
		fmt.Println(err)
	}
}

Steps:

  1. Start NATS:

    docker run --rm --name grain-role-nats -p 4222:4222 nats:2.10-alpine
  2. In three terminals, start the gateway first and then both workers:

    go run . -id gateway -roles gateway -discovery-port 33220 -peers-port 33221 -remoting-port 33222 -activate
    go run . -id worker-1 -roles game-worker -discovery-port 33230 -peers-port 33231 -remoting-port 33232
    go run . -id worker-2 -roles game-worker -discovery-port 33240 -peers-port 33241 -remoting-port 33242
  3. Confirm that role-bound-grain initially logs ACTIVATED on worker-1 or worker-2,
    never on the gateway.

  4. Send SIGTERM to the worker process that logged the activation.

  5. Observe that eager relocation can log the next ACTIVATED message on port 33222,
    even though that node advertises only the gateway role.

Verified output from the sample:

2026/09/03 12:35:23 STARTED id=gateway roles=[gateway] remoting=33222
2026/09/03 12:35:23 STARTED id=worker-1 roles=[game-worker] remoting=33232
2026/09/03 12:35:22 STARTED id=worker-2 roles=[game-worker] remoting=33242
2026/09/03 12:35:31 CREATED identity=main.rolegrain/role-bound-grain
2026/09/03 12:35:31 ACTIVATED grain=role-bound-grain node=127.0.0.1:33242
2026/09/03 12:35:49 DEACTIVATED grain=role-bound-grain node=127.0.0.1:33242
2026/09/03 12:35:49 ACTIVATED grain=role-bound-grain node=127.0.0.1:33222

The initial activation correctly selected worker-2. Immediately after worker-2 received
SIGINT, eager relocation activated the same Grain on the gateway-only node.

The relevant implementation locations at commit
8fd0257885fc84578cc3ae9532f58b4537044903 appear to be:

  • internal/internalpb/grain.proto: the serialized Grain has no activation-role field.
  • actor/relocation_worker.go, allocateGrains: grains are split across the leader and
    all peers without role filtering.
  • actor/grain_engine.go, recreateGrainOnce: reconstructed options do not include
    WithActivationRole.

Expected behavior

The activation role should be retained in Grain relocation metadata. An eager Grain
created with WithActivationRole("game-worker") should be relocated only to surviving
nodes advertising game-worker.

If no eligible node exists, relocation should fail explicitly and include the Grain in a
RelocationFailed event instead of activating it on an ineligible node.

Screenshots

Not applicable. The ACTIVATED log lines include the remoting port and show both the
original eligible worker and the ineligible relocation target.

Library Version:

  • Go-Akt version: v4.5.4-0.20260902210446-8fd0257885fc
  • Go version: go1.27.0 linux/amd64

Additional context

The relocation documentation states that role-constrained actors are assigned only to
eligible nodes. It also states that eager Grains are recreated up front on their allocated
target. Preserving WithActivationRole for Grains would make initial activation and
post-relocation placement consistent.

The issue is reproducible after graceful shutdown. Based on the serialized Grain schema
and the shared relocation allocation path, abrupt-departure recovery appears affected as
well.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions