MClem's Journal for GSoC '25

Week 8: Dynamic State Generation and Typed Templates

14 min read

Week 8: Dynamic State Generation

After implementing decomposition and species management last week, we hit another fundamental limitation: behaviors often require custom states that don’t exist in the base Tölvera system. Asking the LLM to work with only the default particle properties (position, velocity, mass, etc.) limited the types of behaviors we could generate. We have had some great success from prior weeks with effective program generation/synthesis, but there wasn’t necessarily a direct tie into the artificial life portion of the project quite yet. This week’s work focused on building a dynamic state generation system that automatically creates the states a behavior needs, paired with a template-based approach to ensure the generated code actually works. Yes…that’s right… I tried out the jinja2 template approach for this based on some research we found here. After the initial state generation, I had a lot of issues with errors constantly, so I went back to this JSON approach with Pydantic for the states and then the rest were left as typed holes.

The code for this week can be seen in the state-generation-demo branch with the enhanced demo at poe_demo.py.

State Generation Problem

Previously, when a user requested something like “particles move faster during the day and rest at night”, the LLM would either:

  1. Try to hack it using existing states (storing time in unused fields like mass)
  2. Create overly complex workarounds that didn’t really capture the intended behavior
  3. Straight up ignore the request and just pretend it did something :)

What we needed was a way for the system to:

  • Analyze the behavior description and identify required states
  • Create those states dynamically appropriate types and ranges
  • Assert the generated expert code correctly accesses these states (experts grabbing those states)
  • Handle temporal updates for time-based states (the main focus this week)

Updated System Architecture

The updated arch now includes state synthesis as a core component:

graph TD
    A[User Description]:::input --> B[TolveraBehaviorAgent]:::agent
    B --> C[StateSynthesizer]:::state
    C --> D[Analyze State Requirements]:::stateLight
    D --> E[State Specification]:::stateLight
    E --> F[DynamicStateManager]:::state
    F --> G[Create Tölvera States]:::stateLight
    G --> H[PoEExpertSynthesizer]:::synthesis
    H --> I{Behavior Router}:::synthesisLight
    I --> J[Generate Expert with States]:::synthesisLight
    J --> K[Template-Based Kernel Gen]:::synthesisLight
    K --> L[Integration & Compilation]:::output

    classDef input fill:#ff6b6b,stroke:#c92a2a,stroke-width:3px,color:#fff
    classDef agent fill:#4ecdc4,stroke:#0b7285,stroke-width:3px,color:#fff
    classDef state fill:#f783ac,stroke:#e64980,stroke-width:3px,color:#fff
    classDef stateLight fill:#faa2c1,stroke:#f783ac,stroke-width:2px
    classDef synthesis fill:#748ffc,stroke:#4c6ef5,stroke-width:3px,color:#fff
    classDef synthesisLight fill:#91a7ff,stroke:#748ffc,stroke-width:2px
    classDef output fill:#ff8787,stroke:#fa5252,stroke-width:3px,color:#fff

State Synthesis Pipeline

1. State Analysis (StateSynthesizer)

The first step analyzes the natural language description to determine what states are needed:

# From state_synthesizer.py
async def analyze_state_requirements(self, description: str) -> Dict[str, Any]:
    """
    Analyze behavior description and return required states using structured outputs.
 
    Returns:
        Dictionary containing:
        - global_states: Global properties (e.g., day_phase, time)
        - particle_states: Per-particle properties (e.g., energy, age)
        - species_states: Per-species properties
        - temporal_config: Time configuration if needed
    """

The LLM is prompted with the behavior description and returns a structured specification. For “particles move faster during the day and rest at night”, it identifies:

  • Global state: time_of_day (ti.f32, 0.0-1.0) - tracks day/night cycle
  • Particle state: movement_speed (ti.f32, 0.0-100.0) - individual particle speeds
  • Temporal config: day_duration=10.0 seconds
2. Dynamic State Creation (DynamicStateManager)

Once we have the state specification, the DynamicStateManager creates actual Tölvera states:

# From dynamic_state_manager.py
def create_states_from_spec(self, state_spec: Dict[str, Any]) -> Dict[str, str]:
    """
    Create Tölvera states based on the specification from StateSynthesizer.
    """
    created_states = {}
 
    # Create global states
    if 'global_states' in state_spec and state_spec['global_states']:
        state_name = self._create_global_states(state_spec['global_states'])
        created_states['global'] = state_name
 
    # Create particle states
    if 'particle_states' in state_spec and state_spec['particle_states']:
        state_name = self._create_particle_states(state_spec['particle_states'])
        created_states['particle'] = state_name
 
    # Create species states
    if 'species_states' in state_spec and state_spec['species_states']:
        state_name = self._create_species_states(state_spec['species_states'])
        created_states['species'] = state_name
 
    # Same thing for temporal states too...

The manager creates states with proper Taichi types and integrates them into Tölvera’s state system. States are namespaced (e.g., llm_global, llm_particle) to avoid conflicts with built-in states.

This could be an oversight for me because I’m suggesting grouping here and that might not be the best way to handle that…could try calling the LLM to make this too

3. Typed Template Generation

This was a shot in the dark to see if the “typed holes” approach worked well. I didn’t have time to test everything (species information, interaction characteristcs, etc), but here’s what I have right now:

# Template for expert functions
@ti.func
def expert_{{ name }}(pos: ti.math.vec2, vel: ti.math.vec2, mass: ti.f32, species: ti.i32, particle_idx: ti.i32) -> ti.math.vec2:
    # TYPED CONTEXT: State access
    {% for state_access in state_accesses %}
    {{ state_access }}
    {% endfor %}
 
    # TYPED HOLE[ti.math.vec2]: Force calculation
    {{ force_calculation }}
 
    return force

This approach ensures:

  • Correct function signatures
  • Proper state access patterns
  • Variables declared before use
  • Correct return types

This is essentially a list of all the things I was running into during generation so that’s why we went with this :)

Walkthrough

For a demonstration of this since there’s a lot of moving parts, let’s trace through the example “particles move faster during the day and rest at night”:

Step 1: Natural Language Input
# User provides description
description = "particles move faster during the day and rest at night"
Step 2: State Analysis
StateSynthesizer - INFO - Analyzing state requirements
StateSynthesizer - INFO - Structured state analysis successful

The LLM analyzes and returns:

{
  "global_states": {
    "time_of_day": {
      "type": "ti.f32",
      "min": 0.0,
      "max": 1.0,
      "description": "A value between 0.0 (night) and 1.0 (day)"
    }
  },
  "particle_states": {
    "movement_speed": {
      "type": "ti.f32",
      "min": 0.0,
      "max": 100.0,
      "description": "The speed at which the particle moves"
    }
  },
  "temporal_config": {
    "day_duration": 10.0,
    "frame_rate": 60.0
  }
}
Step 3: State Creation
DynamicStateManager - INFO - Created global state 'llm_global' with properties: ['time_of_day']
DynamicStateManager - INFO - Created particle state 'llm_particle' with properties: ['movement_speed']

The manager creates Tölvera states that can be accessed in Taichi kernels.

Step 4: Expert Synthesis with State Context
PoEExpertSynthesizer - INFO - Classifying behavior: 'particles move faster during the day and rest at night'
PoEExpertSynthesizer - INFO - Behavior classified as: SINGLE

The synthesizer generates an expert function with state access:

@ti.func
def expert_day_night(pos: ti.math.vec2, vel: ti.math.vec2, mass: ti.f32, species: ti.i32, particle_idx: ti.i32) -> ti.math.vec2:
    # Access states
    time_of_day = tv.s.llm_global.field[0].time_of_day
    speed_multiplier = tv.s.llm_particle.field[particle_idx].movement_speed
 
    # Initialize force before conditionals
    force = ti.math.vec2(0.0, 0.0)
 
    # Random movement
    angle = ti.random() * 2 * 3.14159
    base_force = ti.math.vec2(ti.cos(angle), ti.sin(angle)) * 150.0
 
    # Scale based on day/night
    if time_of_day > 0.5:  # Day
        force = base_force * speed_multiplier
    else:  # Night
        force = base_force * 0.1
 
    return force
Step 5: State Reference Validation

The system can detect if the wrong state name was used and automatically calls the LLM to try and correct them.

Step 6: Integration Kernel Generation
PoESynthesizer - INFO - Synthesizing integration kernel using template approach

Using templates, the system generates a kernel that properly accesses all states, calls experts with correct parameters, updates particles with accumulated forces, and handles boundary conditions

Step 7: Temporal Update Generation

We needed a way to push the clock forward so to speak for this, so we add an update_temporal_states method if there is something in the temporal states and then we go from there.

@ti.kernel
def update_temporal_states():
    """Update time-based global states."""
    # Define temporal constants from configuration
    frames_per_day = 600.0  # 10 seconds at 60 FPS
 
    # Update time of day
    frame_in_day = tv.frame_count % frames_per_day
    tv.s.llm_global.field[0].time_of_day = frame_in_day / frames_per_day
Step 8: Final Compilation and Execution
PoECore - INFO - Integration kernel successfully regenerated

The complete system is compiled and ready to run (hopefully 🤞).

Demo: Day/Night Particle Behavior

The video shows particles with the “move faster during the day and rest at night” behavior.

Recap

  1. Automatic State Discovery: The LLM analyzes behaviors and identifies needed states without human intervention.

  2. Type-Safe State Creation: States are created with Taichi types and value ranges

  3. Template-Based Code Generation: Using typed holes ensures:

    • Variables always declared before use
    • Correct parameter order in function calls
    • Proper state access patterns
    • No undefined variable references

  4. State Context Propagation: The system maintains state information throughout the pipeline

  5. Temporal State Updates: Time-based behaviors get automatic update kernels that run each frame.

Key Issues We Had

  • State Name Consistency: The LLM would often generate different names for the same state. We now validate and correct references automatically.
  • Declaration Order: LLMs often use variables before declaring them. Templates enforce proper declaration order with the sketch.
  • Type Mismatches: States are created with explicit types, preventing the LLM from treating floats as vectors or vice versa.

Results

The state generation system dramatically expands what behaviors can be expressed (which is great) but this next week will focus mostly on fixing this up and making sure the other aspects of this whole system are robust enough to handle this huge structural change.

What’s Next

While the state generation works well, we have another long week ahead to make sure this is working how we intended. I’m not certain if it’s going to be robust enough to handle the species implementation so a lot of that code will need to be cleaned up before any of it is fully usable.

The foundation is solid though - we can now generate artificial life simulations from natural language descriptions. The LLM handles the complexity while the system ensures the generated code actually works.

Detailed Code Execution Walkthrough

To better understand how all the pieces fit together, let’s trace through exactly what happens when running poe_demo.py with our example behavior:

1. Demo Initialization (poe_demo.py:186-201)
async def demo_simple_behaviors():
    tv_config = {
        "particles": 100,
        "px": "pixels",
        "gpu": "metal" if sys.platform == "darwin" else "cuda"
    }
 
    tv = Tolvera(**tv_config)
    agent = TolveraBehaviorAgent(tv)
    synthesizer = PureLLMSynthesizer(model_name="qwen3:4b", enable_decomposition=False, tolvera_instance=tv)

What happens:

  • Creates Tölvera instance with 100 particles
  • Initializes TolveraBehaviorAgent (poe_integration.py) to orchestrate the synthesis
  • Creates synthesizer with state generation enabled
2. Behavior Addition (poe_demo.py:255-261)
expert = await agent.add_expert_from_description(
    description,
    synthesizer.synthesizer,
    weight=weight,
    use_decomposition=False,
    use_states=True  # Enable state synthesis
)

Calls: TolveraBehaviorAgent.add_expert_from_description() (poe_integration.py:89)

3. State Analysis Phase (poe_integration.py:115-127)
# In add_expert_from_description
if use_states and hasattr(synthesizer, 'synthesize_with_states'):
    result = await synthesizer.synthesize_with_states(
        description, self.tolvera_instance
    )

Calls: PoEExpertSynthesizer.synthesize_with_states() (poe_synthesis.py:456)

4. State Requirements Analysis (poe_synthesis.py:463-473)
# In synthesize_with_states
state_spec = await self.state_synthesizer.analyze_state_requirements(description)

Calls: StateSynthesizer.analyze_state_requirements() (state_synthesizer.py:59)

This sends the structured prompt to the LLM and gets back the state specification.

5. Dynamic State Creation (poe_synthesis.py:481-492)
# Still in synthesize_with_states
if state_spec and any_states:
    created_state_names = self.state_manager.create_states_from_spec(state_spec)

Calls: DynamicStateManager.create_states_from_spec() (dynamic_state_manager.py:51)

This creates the actual Tölvera states:

  • _create_global_states() → Creates tv.s.llm_global with time_of_day field
  • _create_particle_states() → Creates tv.s.llm_particle with movement_speed field
6. Expert Function Generation (poe_synthesis.py:495-510)
# Generate expert with state context
state_context = self.state_manager.get_synthesis_context()
# ... behavior classification ...
function_result = await self._generate_single_particle_expert(
    description, existing_names, state_context
)

Calls: PoEExpertSynthesizer._generate_single_particle_expert() (poe_synthesis.py:665)

The LLM generates the expert function with access to the state documentation.

7. Error Detection and Correction (poe_synthesis.py:726-751)
# Validate and fix state references
invalid_refs = self._validate_state_references(code, state_context)
if invalid_refs:
    code = await self._fix_invalid_state_references(
        code, invalid_refs, state_context, description
    )

The system detects if the LLM used wrong state names and corrects them.

8. Kernel Accumulation (poe_synthesis.py:797-802)
# Save successful kernel
if self.kernel_accumulator:
    uuid = self.kernel_accumulator.save_kernel(
        final_code, metadata
    )

Calls: KernelAccumulator.save_kernel() → Saves to kernels_repository.py

9. Integration Kernel Generation (poe_integration.py:163-166)
# Back in add_expert_from_description
self.poe_system.regenerate_integration_kernel(synthesizer)

Calls: PoEBehaviorSystem.regenerate_integration_kernel() (poe_core.py:234)

10. Template-Based Kernel Synthesis (poe_synthesis.py:1053-1089)
# In synthesize_integration_kernel_template
template = self.template_env.get_template('integration_kernel.j2')
# Get kernel configuration from LLM
config = await self._get_kernel_configuration(expert_info)
# Render template
kernel_code = template.render(config)

The template ensures proper structure while the LLM provides the configuration.

11. Temporal Update Generation (poe_demo.py:295-302)
# Generate temporal update code if needed
update_code = await synthesizer.synthesizer.state_synthesizer.generate_state_update_code(
    state_spec, temporal_config, behavior_desc
)

Calls: StateSynthesizer.generate_state_update_code() (state_synthesizer.py:343)

12. Final Sketch Generation (poe_demo.py:304)
filename = save_generated_sketch_to_file(agent, tv_config, state_update_code=state_update_code)

This creates the final Python file.

Final Code

And here’s an example of the full final generated code from this natural language:

"""
 
Dynamically generated Tölvera sketch.
"""
import taichi as ti
from tolvera import Tolvera, run
import numpy as np
from math import pi
 
def main(**kwargs):
    tv = Tolvera(**kwargs)
 
    # Species configuration: [0]
    species_map = ti.field(dtype=ti.i32, shape=1)
    species_map[0] = 0
    @ti.kernel
    def init_particles():
        for i in range(tv.pn):
            tv.p.field[i].active = 1.0
            tv.p.field[i].pos = ti.Vector([ti.random() * tv.x, ti.random() * tv.y])
            tv.p.field[i].vel = ti.Vector([0.0, 0.0])
            tv.p.field[i].size = 5.0
            tv.p.field[i].mass = 1.0
            # Assign species using the mapping
            species_index = i % 1
            tv.p.field[i].species = species_map[species_index]
    init_particles()
    tv.s.species.field[0].rgba = [1.0, 0.3, 0.3, 1.0]
 
    # Creating dynamic states
 
    # Particle states: ['energy'] - Higher minimum energy
    tv.s.llm_particle = {
        "state": {
            "energy": (ti.f32, 20.0, 500.0),  # Much higher minimum energy
        },
        "shape": tv.pn,
        "osc": ("get"),
        "randomise": True
    }
 
    # Global day/night multiplier
    day_night_multiplier = ti.field(dtype=ti.f32, shape=())
    day_night_multiplier[None] = 1.0
 
    # ***** Temporal State Update *****
    @ti.kernel
    def update_temporal_states():
        """Update time-based particle states with cyclical day/night behavior."""
        # Define temporal constants from configuration
        frames_per_day = 600.0  # Use actual value from temporal config
        day_duration = 10.0  # Use actual value from temporal config
        time_scale = 1.0  # Use actual value from temporal config
 
        # CRITICAL: Initialize all variables BEFORE conditionals
        current_frame = tv.ctx.i[None]
        time_of_day = 0.0
        energy_change = 0.0
        activity_multiplier = 0.0
 
        # Calculate time_of_day based on current frame
        if frames_per_day > 0:
            time_of_day = (current_frame % frames_per_day) / frames_per_day
 
        # Calculate day/night activity multiplier - NEVER go to zero
        # Range from 0.4 (night) to 1.0 (day) so movement never stops
        activity_multiplier = 0.4 + 0.6 * ((ti.cos(time_of_day * 2 * 3.14159) + 1.0) * 0.5)
 
        # Store global multiplier for use in force calculations
        day_night_multiplier[None] = activity_multiplier
 
        # Update particle states with cyclical energy behavior
        for i in range(tv.pn):
            # Initialize variables before conditionals
            energy = tv.s.llm_particle.field[i].energy
            energy_change = 0.0
 
            # CYCLICAL energy behavior with more dramatic changes
            if activity_multiplier > 0.7:  # DAY: energy increases
                energy_change = 2.0 * activity_multiplier  # Faster energy gain
            else:  # NIGHT: energy decreases
                energy_change = -1.0 * (1.0 - activity_multiplier)  # Slower energy loss
 
            # Apply energy change
            energy += energy_change
 
            # Clamp energy to reasonable bounds with higher minimum
            if energy > 100.0:
                energy = 100.0
            elif energy < 50.0:  # Much higher minimum energy
                energy = 50.0
 
            # Update particle energy
            tv.s.llm_particle.field[i].energy = energy
 
    # ***** Generated Expert Functions *****
    @ti.func
    def expert_energy_loss(pos: ti.math.vec2, vel: ti.math.vec2, mass: ti.f32, species: ti.i32, particle_idx: ti.i32) -> ti.math.vec2:
        # Initialize force before any conditionals
        force = ti.math.vec2(0.0, 0.0)
 
        # Get energy from the particle state
        energy = tv.s.llm_particle.field[particle_idx].energy
 
        # FIXED: Higher energy means MORE movement with stronger base force
        energy_factor = energy / 10.0  # Range 0.5 to 1.0 now
 
        # Random movement component - MUCH stronger base force
        angle = ti.random() * 2 * 3.14159
        base_force = ti.math.vec2(ti.cos(angle), ti.sin(angle)) * 1500.0  # Increased from 400 to 1500
 
        # Apply energy factor to force
        force = base_force * energy_factor
 
        # Apply day/night multiplier (now ranges 0.4-1.0, never zero)
        force *= day_night_multiplier[None]
 
        # Simplified: just one extra boost for very active periods
        if day_night_multiplier[None] > 0.9:  # Peak day only
            force *= 1.3
 
        return force
 
    # ***** Generated Integration Kernel *****
 
 
    @ti.kernel
    def apply_all_experts():
        """Main kernel that integrates all expert forces including interactions."""
        dt = 0.016  # ~60fps timestep
 
        for i in range(tv.pn):
            if tv.p.field[i].active > 0:
                # TYPED CONTEXT: Particle state
                pos = tv.p.field[i].pos  # type: ti.math.vec2
                vel = tv.p.field[i].vel  # type: ti.math.vec2
                mass = tv.p.field[i].mass  # type: ti.f32
                species = tv.p.field[i].species  # type: ti.i32
 
                # TYPED HOLE[ti.math.vec2]: Initialize total force
                total_force = ti.math.vec2(0.0, 0.0)
 
 
 
                # === SINGLE-PARTICLE FORCES ===
 
                # Apply single-particle experts
                total_force += expert_energy_loss(pos, vel, mass, species, i) * 1.00
 
 
 
 
                # TYPED OPERATIONS: Physics update with stronger integration
                # Much lighter damping and stronger dt for more responsive movement
                damping = 0.90 if day_night_multiplier[None] > 0.7 else 0.85  # Much lighter damping
                tv.p.field[i].vel = tv.p.field[i].vel * damping + total_force * dt
 
                # Update position based on velocity
                new_pos = tv.p.field[i].pos + tv.p.field[i].vel * dt
 
 
                # Boundary handling
                # Bounce off boundaries
                if new_pos[0] < 0:
                    new_pos[0] = -new_pos[0]
                    tv.p.field[i].vel[0] = -tv.p.field[i].vel[0] * 0.8  # Energy loss
                elif new_pos[0] > tv.x:
                    new_pos[0] = 2 * tv.x - new_pos[0]
                    tv.p.field[i].vel[0] = -tv.p.field[i].vel[0] * 0.8
 
                if new_pos[1] < 0:
                    new_pos[1] = -new_pos[1]
                    tv.p.field[i].vel[1] = -tv.p.field[i].vel[1] * 0.8
                elif new_pos[1] > tv.y:
                    new_pos[1] = 2 * tv.y - new_pos[1]
                    tv.p.field[i].vel[1] = -tv.p.field[i].vel[1] * 0.8
 
                tv.p.field[i].pos = new_pos
 
 
 
    @tv.render
    def _():
        tv.px.diffuse(0.99)
 
        tv.p()
 
        # Update temporal states
        update_temporal_states()
 
        apply_all_experts()
 
        tv.px.particles(tv.p, tv.s.species())
 
        return tv.px
 
if __name__ == "__main__":
    run(main)
Key Files
  • poe_demo.py: Entry point and orchestration
  • poe_integration.py: High-level behavior agent that coordinates the process
  • state_synthesizer.py: Analyzes descriptions to determine state requirements
  • dynamic_state_manager.py: Creates actual Tölvera states from specifications
  • poe_synthesis.py: Generates expert functions and integration kernels
  • poe_ollama.py: Handles LLM communication
  • taichi_error_detector.py: Detects common code generation errors
  • state_code_generator.py: Generates state initialization code
  • Templates (in llm/templates/):
    • expert_template.j2: Template for expert functions
    • integration_kernel.j2: Template for the main kernel

The entire flow from natural language to working simulation involves orchestration of state analysis, dynamic creation, code generation with templates, error correction, and compilation - all working together to produce reliable, particle simulations. It’s starting to get very complex here, but we’re on to something with this approach! Stoked with how this turned out and excited to see where this goes in the coming days!

Graph View

Backlinks