Emerge is a decentralized synchronization algorithm that enables independent agents to coordinate their behavior without central control. Think of it like fireflies synchronizing their flashing - no firefly is in charge, but they all end up flashing together.
Schedulers tell you exactly when to run (like cron: "run at 3pm").
Emerge helps you figure out when to run together (like "let's all batch our API calls when we're ready").
Emerge is dynamic and adaptive, while schedulers are static and predetermined.
No! The client API is simple:
client := emerge.MinimizeAPICalls(scale.Medium)
client.Start(ctx)The complex math is handled internally. You just specify what you want to achieve.
No. Consensus algorithms help you agree on a single value (like "who is the leader?"). Emerge helps you synchronize continuous behavior (like "when should we all act?").
- Raft: "Let's vote on value X"
- Emerge: "Let's coordinate our timing"
See Alternatives for detailed comparisons.
For a comprehensive guide on getting started with emerge, including:
- How to choose the right scale
- How to select the appropriate goal
- Understanding convergence times
- Integrating with existing systems
→ See the Quick Start Guide
Start small and scale up:
- Testing: Use Tiny (20 agents)
- Production start: Use Small (50 agents) or Medium (200 agents)
- Scale as needed: Move to Large (1000) or Huge (2000+) when ready
See Scales for detailed configurations and resource requirements.
// Start small
client := emerge.MinimizeAPICalls(scale.Small)
// Scale up later
client := emerge.MinimizeAPICalls(scale.Large)Ask yourself what you're trying to optimize:
- Want to batch operations? →
MinimizeAPICalls - Want to spread load? →
DistributeLoad - Want agreement? →
ReachConsensus - Want speed? →
MinimizeLatency - Want to save resources? →
SaveEnergy
See Goals for detailed descriptions and Use Cases for real-world examples.
Convergence time depends on many factors:
- Scale - More agents generally take longer
- Goal - Different goals have different convergence characteristics
- Network topology - Full mesh converges faster than sparse networks
- Initial conditions - Random start vs partially synchronized
- Parameters - Coupling strength, update frequency, etc.
Rough estimates (actual times vary):
- Tiny (20 agents): Seconds
- Small (50 agents): Several seconds
- Medium (200 agents): Tens of seconds
- Large (1000 agents): Up to minutes
- Huge (2000+ agents): Several minutes
These are approximations. Always test with your specific configuration and workload to determine actual convergence times.
Yes! Emerge is designed to integrate with existing systems without major rewrites. Here's how:
Step 1: Add emerge client to your service
// In your existing service
type MyService struct {
// Your existing fields stay the same
database *sql.DB
cache *redis.Client
// Add emerge client
emergeClient *emerge.Client
}Step 2: Initialize emerge alongside your existing setup
func NewMyService() *MyService {
s := &MyService{
database: connectDB(),
cache: connectRedis(),
// Add emerge with appropriate goal
emergeClient: emerge.MinimizeAPICalls(scale.Small),
}
// Start emerge in background
go s.emergeClient.Start(context.Background())
return s
}Step 3: Use emerge to coordinate existing operations
type MyService struct {
// ... existing fields ...
emergeClient *emerge.Client
pendingItems []Item // Add a field to accumulate items
mu sync.Mutex
}
// Your existing method now accumulates items
func (s *MyService) ProcessItem(item Item) {
s.mu.Lock()
s.pendingItems = append(s.pendingItems, item)
s.mu.Unlock()
}
// Add a background goroutine to handle batching
func (s *MyService) RunBatchProcessor(ctx context.Context) {
ticker := time.NewTicker(100 * time.Millisecond)
defer ticker.Stop()
wasConverged := false
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
isConverged := s.emergeClient.IsConverged()
// Batch when we transition into converged state
// This prevents batching continuously while converged
if isConverged && !wasConverged {
s.mu.Lock()
if len(s.pendingItems) > 0 {
// Now synchronized - batch all pending items
s.database.BatchInsert(s.pendingItems)
s.pendingItems = nil
// Items will accumulate again until next convergence
}
s.mu.Unlock()
}
wasConverged = isConverged
}
}
}The key is that emerge doesn't replace your existing logic - it helps coordinate WHEN batched operations happen. Items accumulate until emerge says "now we're all synchronized, safe to batch!"
Tested up to 2000 agents (Huge scale). Theoretically can handle more, but you'll need to:
- Increase memory (roughly 5KB per agent)
- Adjust parameters for larger scales
- Consider hierarchical organization for 10,000+ agents
No, emerge can work in-process with goroutines. For distributed systems, you'll need to implement neighbor communication, but emerge doesn't mandate any specific network protocol.
Emerge is resilient to failures:
- Up to 50% of agents can fail without breaking synchronization
- Remaining agents continue coordinating
- No leader election or recovery protocol needed
- System self-heals as new agents join
The convergence is deterministic (will always reach the goal), but the exact path and time to convergence may vary based on initial conditions and random factors.
While agents should follow the same protocol, they can have:
- Different natural frequencies
- Different stubbornness levels
- Different energy constraints
- Different workloads
The key is they all participate in the same synchronization protocol.
It's not about speed, it's about coordination:
- Message queue: Handles message delivery
- Emerge: Coordinates when to send messages
They solve different problems. You can use both together:
if emerge.IsConverged() {
queue.PublishBatch(messages) // Use both!
}Minimal overhead:
- Memory: ~5KB per agent
- CPU: ~50ns per agent update (atomic operations)
- Network: Only neighbor communication (not all-to-all)
The coordination benefits usually outweigh the overhead.
Better than linear for many operations:
- Communication: O(k) where k = neighbors (constant, not O(N))
- Convergence time: O(log N) in many cases
- Memory: O(N) - linear with agent count
- Choose the right scale - Don't use Huge if Medium works
- Select appropriate goals - Match goal to use case
- Use proper patterns - High-frequency for batching, sparse for energy saving
- Monitor coherence - Don't over-synchronize
Common causes:
- Coupling too weak - Increase coupling strength
- Network partitioned - Check agent connectivity
- Energy depleted - Increase recovery rate
- Wrong goal - Verify goal matches your needs
Usually means:
- Coupling strength too high (agents over-correcting)
- Energy depletion cycles (agents run out of energy)
- Conflicting influences (check network topology)
Solution: Reduce coupling strength or increase energy recovery.
Could be:
- Scale too large for the goal
- Pattern doesn't match goal (e.g., sparse pattern with batching goal)
- Natural frequencies too diverse
- Network topology too sparse
No, and you shouldn't try. Emerge is about emergent coordination. Forcing it defeats the purpose and benefits. If you need immediate coordination, consider a different tool.
No. Timers are static and predetermined. Emerge dynamically adapts to system conditions, load, failures, and other factors. It's like comparing a sundial to a smart watch.
Database locks provide mutual exclusion (one at a time). Emerge provides coordination (all together). They solve different problems:
- Lock: "Only I can access this"
- Emerge: "Let's all act together"
The complexity is hidden. Using emerge is as simple as:
client := emerge.MinimizeAPICalls(scale.Small)
client.Start(ctx)
if client.IsConverged() {
// Do your thing
}That's simpler than implementing your own coordination logic.
Agents need to follow the same synchronization protocol, but can have:
- Different workloads
- Different processing speeds
- Different resource constraints
- Different business logic
No. Use emerge when you need:
- Distributed coordination without central control
- Adaptive synchronization
- Resilience to failures
- Scalable coordination
Don't use emerge for:
- Simple task distribution (use work queues)
- Fixed scheduling (use cron)
- Mutual exclusion (use locks)
- Message passing (use message queues)
- Monitor coherence - Track synchronization level
- Check energy levels - Ensure agents have resources
- Verify topology - Confirm agents can see neighbors
- Use small scale - Test with Tiny scale first
- Enable logging - Add coherence/phase logging
func TestWithEmerge(t *testing.T) {
// Use tiny scale for tests
client := emerge.MinimizeAPICalls(scale.Tiny)
// Use shorter timeouts
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
client.Start(ctx)
// Wait for convergence
require.Eventually(t, client.IsConverged, 5*time.Second, 100*time.Millisecond)
}Yes, but allow time for reconvergence:
// Start with one goal
client := emerge.MinimizeAPICalls(scale.Medium)
client.Start(ctx)
// Later, switch goals
client.Stop()
client = emerge.DistributeLoad(scale.Medium)
client.Start(ctx)- Simulation:
simulations/emerge/- Interactive demo - Tests:
emerge/swarm/*_test.go- Test cases - Documentation:
docs/emerge/- Detailed guides
- Check this FAQ first
- Search existing issues on GitHub
- Provide minimal reproduction case
- Include coherence logs and agent counts
If X involves coordinating multiple independent entities without central control, probably yes! Emerge is quite flexible. Check if one of the existing goals matches your needs, or consider combining emerge with other tools.
Yes, the emerge primitive is production-ready. It has:
- Comprehensive test coverage
- Performance optimizations
- Proven algorithm (Kuramoto model)
- Resilience to failures
Always test with your specific use case and scale before production deployment.
- Quick Start Guide - Comprehensive getting started guide
- Getting Started - High-level overview
- Algorithm - How emerge works
- Goal-Directed - How emerge pursues goals
- Disruption - Handling failures
- Use Cases - Real-world applications
- Glossary - Term definitions