update experiments

Author: HenriqueAssumpcao

README.md

@@ -1,20 +1,13 @@
 # Experiments with CodeEvolve
-This repository contains benchmark implementations, experimental configurations, and reproducibility code for the CodeEvolve paper:
+This repository contains benchmark implementations, experimental configurations, and reproducibility code for the CodeEvolve paper (versions 1 and 2):
 
 > **CodeEvolve: an open source evolutionary coding agent for algorithm discovery and optimization**  
 > Henrique Assumpção, Diego Ferreira, Leandro Campos, Fabricio Murai  
 > [arXiv:2510.14150](https://arxiv.org/abs/2510.14150)
 
-## Overview
-
-TODO
-
-## Repository Structure
-
-TODO
 
 ## Prerequisites
-Install CodeEvolve and dependencies:
+Install CodeEvolve v0.1.0 and dependencies:
 
 ```bash
 # Clone and install CodeEvolve framework
@@ -33,10 +26,6 @@ export API_KEY=your_api_key
 export API_BASE=your_api_base_url
 ```
 
-## Reproducing results
-
-TODO
-
 ## Citation
 
 If you use CodeEvolve in your research, please cite our paper:

problems/alphaevolve_math_problems/circle_packing_rect/configs/config.yaml

@@ -0,0 +1,105 @@
+SYS_MSG: |
+  SETTING:
+  You are an expert computational geometer and optimization specialist with deep expertise in circle packing problems, geometric optimization algorithms, and constraint satisfaction.
+  Your mission is to evolve and optimize a constructor function that generates an optimal arrangement of exactly 21 non-overlapping circles within a rectangle, maximizing the sum of their radii.
+
+  PROBLEM CONTEXT:
+  - **Objective**: Create a function that returns optimal (x, y, radius) coordinates for 21 circles
+  - **Benchmark**: Beat the AlphaEvolve state-of-the-art result of sum_radii = 2.3658321334167627
+  - **Container**: Rectangle with perimeter = 4 (width + height = 2). You may choose optimal width/height ratio
+  - **Constraints**: 
+    * All circles must be fully contained within rectangle boundaries
+    * No circle overlaps (distance between centers ≥ sum of their radii)
+    * Exactly 21 circles required
+    * All radii must be positive
+
+  COMPUTATIONAL RESOURCES & IMPLEMENTATION GUIDELINES:
+  **Core packages**: numpy, scipy, sympy, pandas, networkx, jax, torch, numba, scikit-learn
+
+  **Additional useful packages**:
+  - **Global optimization**: `deap` (evolutionary computation), `platypus` (NSGA-II)
+  - **Metaheuristics**: `scikit-opt` (PSO, GA, SA), `nevergrad` (gradient-free optimization), `optuna` (hyperparameter optimization)
+  - **Geometric computing**: `shapely` (geometric operations), `rtree` (spatial indexing), `scipy.spatial` (distance matrices, Voronoi)
+  - **Constraint programming**: `python-constraint`, `ortools` (Google OR-Tools), `cvxpy` (convex optimization)
+  - **Physics engines**: `pymunk` (2D rigid body physics), `Box2D` (collision detection)
+  - **Parallel computing**: `joblib` (embarrassingly parallel), `multiprocessing`, `concurrent.futures`
+  - **Performance**: `cython` (C extensions), `numexpr` (fast numerical expressions)
+
+  PERFORMANCE METRICS:
+  1. **sum_radii**: Total sum of all 21 circle radii (PRIMARY OBJECTIVE - maximize)
+  2. **benchmark_ratio**: sum_radii / 2.3658321334167627 (progress toward beating benchmark)  
+  3. **eval_time**: Execution time in seconds (keep reasonable, prefer accuracy over speed)
+
+  TECHNICAL REQUIREMENTS:
+  - **Determinism**: Use fixed random seeds if employing stochastic methods for reproducibility
+  - **Error handling**: Graceful handling of optimization failures or infeasible configurations
+  - **Memory efficiency**: Avoid excessive memory allocation for distance matrix computations
+  - **Scalability**: Design with potential extension to different circle counts in mind
+
+  # PROMPT-BLOCK-START
+  **Recommended implementation patterns**:
+    - **Hierarchical optimization**: First optimize rectangle aspect ratio, then circle placement within optimal container
+    - **Multi-stage approach**: Start with uniform radii placement, then allow radius variation in second optimization phase
+    - **Constraint violation handling**: Use penalty methods or constraint satisfaction to ensure feasibility
+    - **Parallel evaluation**: Leverage multiple CPU cores for population-based algorithms (GA, PSO, differential evolution)
+    - **Adaptive discretization**: Begin with coarse grid-based positioning, progressively refine to continuous coordinates
+    - **Hybrid algorithms**: Combine global metaheuristics with local gradient-based refinement (scipy.optimize.minimize)
+    - **Restart mechanisms**: Multiple independent runs with different initializations, select best result
+    - **Progressive complexity**: Start with simpler sub-problems (fewer circles) and incrementally add complexity
+    
+    MATHEMATICAL CONSIDERATIONS:
+    - **Aspect ratio optimization**: Golden ratio (~1.618) often emerges as optimal for packing problems, but verify empirically
+    - **Circle size distribution**: Consider both uniform radii and variable radii strategies - sometimes heterogeneous sizes pack more efficiently
+    - **Geometric bounds**: Maximum possible radius for any circle is min(width, height)/2; total area constraint: π∑r² ≤ width×height
+    - **Distance constraints**: For circles i,j: √((xi-xj)² + (yi-yj)²) ≥ ri + rj (non-overlap)
+    - **Boundary constraints**: ri ≤ xi ≤ width-ri and ri ≤ yi ≤ height-ri (containment)
+    - **Symmetry exploitation**: Look for symmetric arrangements (reflection, rotation) that might be optimal
+    - **Packing density**: Theoretical maximum density for infinite plane is π/(2√3) ≈ 0.9069; rectangle constraints reduce this significantly
+    - **Contact graph theory**: Model circle tangencies as graph structures to identify promising geometric configurations
+    
+    ALGORITHMIC STRATEGIES TO CONSIDER:
+    - **Physics-based simulation**: Use pymunk/Box2D with repulsive forces to naturally separate overlapping circles
+    - **Evolutionary algorithms**: NSGA-II or CMA-ES with population size 50-200, run for 1000+ generations
+    - **Simulated annealing**: Temperature scheduling from high (accept worse solutions) to low (hill-climbing)
+    - **Particle swarm optimization**: Velocity-based updates with social/cognitive parameters, include constraint handling
+    - **Sequential placement**: Greedy algorithms that place circles one-by-one in locally optimal positions
+    - **Space partitioning**: Quadtree or spatial hashing for efficient collision detection during optimization
+    - **Gradient-free methods**: Nelder-Mead, Powell, or COBYLA for local refinement of promising configurations
+    - **Multi-objective optimization**: Balance sum_radii maximization with constraint violation minimization
+    - **Clustering approaches**: Group circles into regions, optimize within regions, then optimize region arrangement
+    - **Machine learning guidance**: Train surrogate models to predict promising initial configurations
+    
+    VALIDATION FRAMEWORK:
+    - **Constraint verification**: Automated checking of all geometric constraints (containment, non-overlap)
+    - **Floating-point precision**: Use tolerance ε ≈ 1e-10 for numerical comparisons to handle rounding errors
+    - **Visual validation**: Generate matplotlib plots showing circle arrangements for manual inspection
+    - **Statistical analysis**: Run multiple seeds (≥10) and report mean, std, min, max of sum_radii across runs
+    - **Benchmark comparison**: Calculate improvement percentage over AlphaEvolve baseline
+    - **Regression testing**: Ensure new approaches don't degrade performance on simpler test cases
+    - **Performance profiling**: Use cProfile to identify computational bottlenecks and optimize critical paths
+    - **Edge case testing**: Verify behavior with extreme aspect ratios, very small/large rectangles
+    - **Incremental validation**: Test with fewer circles (5, 10, 15) to build confidence in algorithmic approach
+    - **Cross-validation**: Compare results across different optimization algorithms to identify consistent solutions
+    # PROMPT-BLOCK-END
+      
+CODEBASE_PATH: 'src/'
+INIT_FILE_DATA: {filename: 'init_program.py', language: 'python'}
+EVAL_FILE_NAME: 'evaluate.py'
+EVAL_TIMEOUT: 360
+
+MAX_MEM_BYTES: 1000000000
+MEM_CHECK_INTERVAL_S: 0.1
+
+EVOLVE_CONFIG: {fitness_key: 'benchmark_ratio',
+                num_epochs: 200,ckpt: 5,max_size: 40,init_pop: 6,
+                exploration_rate: 0.3, 
+                selection_policy: 'roulette', selection_kwargs: {roulette_by_rank: True},
+                early_stopping_rounds: 100,
+                num_islands: 5, migration_topology: 'ring', migration_interval: 40, migration_rate: 0.1,
+                meta_prompting: True, num_inspirations: 3,
+                max_chat_depth: 3}
+
+ENSEMBLE: [{model_name: 'GOOGLE_GEMINI-2.5-FLASH', temp: 0.7, top_p: 0.95, retries: 3, weight: 0.8, verify_ssl: False},
+           {model_name: 'GOOGLE_GEMINI-2.5-PRO', temp: 0.7, top_p: 0.95, retries: 3, weight: 0.2, verify_ssl: False}]
+
+SAMPLER_AUX_LM : {model_name: 'GOOGLE_GEMINI-2.5-FLASH', temp: 0.7, top_p: 0.95, retries: 3, weight: 0.8, verify_ssl: False}

problems/alphaevolve_math_problems/circle_packing_rect/configs/config_nocontext.yaml

@@ -0,0 +1,75 @@
+SYS_MSG: |
+  SETTING:
+  You are an expert computational geometer and optimization specialist with deep expertise in circle packing problems, geometric optimization algorithms, and constraint satisfaction.
+  Your mission is to evolve and optimize a constructor function that generates an optimal arrangement of exactly 21 non-overlapping circles within a rectangle, maximizing the sum of their radii.
+
+  PROBLEM CONTEXT:
+  - **Objective**: Create a function that returns optimal (x, y, radius) coordinates for 21 circles
+  - **Benchmark**: Beat the AlphaEvolve state-of-the-art result of sum_radii = 2.3658321334167627
+  - **Container**: Rectangle with perimeter = 4 (width + height = 2). You may choose optimal width/height ratio
+  - **Constraints**: 
+    * All circles must be fully contained within rectangle boundaries
+    * No circle overlaps (distance between centers ≥ sum of their radii)
+    * Exactly 21 circles required
+    * All radii must be positive
+
+  COMPUTATIONAL RESOURCES & IMPLEMENTATION GUIDELINES:
+  **Core packages**: numpy, scipy, sympy, pandas, networkx, jax, torch, numba, scikit-learn
+
+  **Additional useful packages**:
+  - **Global optimization**: `dea[]` (evolutionary computation), `platypus` (NSGA-II)
+  - **Metaheuristics**: `scikit-opt` (PSO, GA, SA), `nevergrad` (gradient-free optimization), `optuna` (hyperparameter optimization)
+  - **Geometric computing**: `shapely` (geometric operations), `rtree` (spatial indexing), `scipy.spatial` (distance matrices, Voronoi)
+  - **Constraint programming**: `python-constraint`, `ortools` (Google OR-Tools), `cvxpy` (convex optimization)
+  - **Physics engines**: `pymunk` (2D rigid body physics), `Box2D` (collision detection)
+  - **Parallel computing**: `joblib` (embarrassingly parallel), `multiprocessing`, `concurrent.futures`
+  - **Performance**: `cython` (C extensions), `numexpr` (fast numerical expressions)
+
+  PERFORMANCE METRICS:
+  1. **sum_radii**: Total sum of all 21 circle radii (PRIMARY OBJECTIVE - maximize)
+  2. **benchmark_ratio**: sum_radii / 2.3658321334167627 (progress toward beating benchmark)  
+  3. **eval_time**: Execution time in seconds (keep reasonable, prefer accuracy over speed)
+
+  TECHNICAL REQUIREMENTS:
+  - **Determinism**: Use fixed random seeds if employing stochastic methods for reproducibility
+  - **Error handling**: Graceful handling of optimization failures or infeasible configurations
+  - **Memory efficiency**: Avoid excessive memory allocation for distance matrix computations
+  - **Scalability**: Design with potential extension to different circle counts in mind
+
+  # PROMPT-BLOCK-START
+
+  **Recommended implementation patterns**:
+  TODO
+
+  MATHEMATICAL CONSIDERATIONS:
+  TODO
+
+  ALGORITHMIC STRATEGIES TO CONSIDER:
+  TODO
+
+  VALIDATION FRAMEWORK:
+  TODO
+
+  # PROMPT-BLOCK-END
+      
+CODEBASE_PATH: 'src/'
+INIT_FILE_DATA: {filename: 'init_program.py', language: 'python'}
+EVAL_FILE_NAME: 'evaluate.py'
+EVAL_TIMEOUT: 360
+
+MAX_MEM_BYTES: 1000000000
+MEM_CHECK_INTERVAL_S: 0.1
+
+EVOLVE_CONFIG: {fitness_key: 'benchmark_ratio',
+                num_epochs: 150,ckpt: 5,max_size: 40,init_pop: 6,
+                exploration_rate: 0.3, 
+                selection_policy: 'roulette', selection_kwargs: {roulette_by_rank: True},
+                early_stopping_rounds: 100,
+                num_islands: 5, migration_topology: 'ring', migration_interval: 40, migration_rate: 0.1,
+                meta_prompting: True, num_inspirations: 3,
+                max_chat_depth: 3}
+
+ENSEMBLE: [{model_name: 'GOOGLE_GEMINI-2.5-FLASH', temp: 0.7, top_p: 0.95, retries: 3, weight: 0.8, verify_ssl: False},
+           {model_name: 'GOOGLE_GEMINI-2.5-PRO', temp: 0.7, top_p: 0.95, retries: 3, weight: 0.2, verify_ssl: False}]
+
+SAMPLER_AUX_LM : {model_name: 'GOOGLE_GEMINI-2.5-FLASH', temp: 0.7, top_p: 0.95, retries: 3, weight: 0.8, verify_ssl: False}

problems/alphaevolve_math_problems/circle_packing_rect/input/evaluate.py

@@ -0,0 +1,126 @@
+# ===--------------------------------------------------------------------------------------===#
+#
+# Part of the CodeEvolve Project, under the Apache License v2.0.
+# See https://github.com/inter-co/science-codeevolve/blob/main/LICENSE for license information.
+# SPDX-License-Identifier: Apache-2.0
+#
+# ===--------------------------------------------------------------------------------------===#
+#
+# This file implements the evaluator for the circle packing problem on a rectangle
+# of perimeter 4.
+#
+# ===--------------------------------------------------------------------------------------===#
+#
+# Some of the code in this file is adapted from:
+#
+# google-deepmind/alphaevolve_results:
+# Licensed under the Apache License v2.0.
+#
+# ===--------------------------------------------------------------------------------------===#
+
+import time
+import numpy as np
+import sys
+import json
+import os
+from importlib import __import__
+
+BENCHMARK = 2.3658321334167627
+NUM_CIRCLES = 21
+TOL = 1e-6
+
+
+def minimum_circumscribing_rectangle(circles: np.ndarray):
+    """Returns the width and height of the minimum circumscribing rectangle.
+
+    Args:
+    circles: A numpy array of shape (num_circles, 3), where each row is of the
+        form (x, y, radius), specifying a circle.
+
+    Returns:
+    A tuple (width, height) of the minimum circumscribing rectangle.
+    """
+    min_x = np.min(circles[:, 0] - circles[:, 2])
+    max_x = np.max(circles[:, 0] + circles[:, 2])
+    min_y = np.min(circles[:, 1] - circles[:, 2])
+    max_y = np.max(circles[:, 1] + circles[:, 2])
+    return max_x - min_x, max_y - min_y
+
+
+def validate_packing_radii(radii: np.ndarray) -> None:
+    n = len(radii)
+    for i in range(n):
+        if radii[i] < 0:
+            raise ValueError(f"Circle {i} has negative radius {radii[i]}")
+        elif np.isnan(radii[i]):
+            raise ValueError(f"Circle {i} has nan radius")
+
+
+def validate_packing_overlap_wtol(circles: np.ndarray, tol: float = 1e-6) -> None:
+    n = len(circles)
+    for i in range(n):
+        for j in range(i + 1, n):
+            dist = np.sqrt(np.sum((circles[i, :2] - circles[j, :2]) ** 2))
+            if dist < circles[i, 2] + circles[j, 2] - tol:
+                raise ValueError(
+                    f"Circles {i} and {j} overlap: dist={dist}, r1+r2={circles[i,2]+circles[j,2]}"
+                )
+
+
+def validate_packing_inside_rect_wtol(circles: np.array, tol: float = 1e-6) -> None:
+    width, height = minimum_circumscribing_rectangle(circles)
+    if width + height > (2 + tol):
+        raise ValueError("Circles are not contained inside a rectangle of perimeter 4.")
+
+
+def evaluate(program_path: str, results_path: str = None) -> None:
+    abs_program_path = os.path.abspath(program_path)
+    program_dir = os.path.dirname(abs_program_path)
+    module_name = os.path.splitext(os.path.basename(program_path))[0]
+
+    circles = None
+    eval_time = 0
+    try:
+        sys.path.insert(0, program_dir)
+        program = __import__(module_name)
+
+        start_time = time.time()
+        circles = program.circle_packing21()
+        end_time = time.time()
+        eval_time = end_time - start_time
+    except Exception as err:
+        raise err
+    finally:
+        if program_dir in sys.path:
+            sys.path.remove(program_dir)
+
+    if not isinstance(circles, np.ndarray):
+        circles = np.array(circles)
+
+    if circles.shape != (NUM_CIRCLES, 3):
+        raise ValueError(
+            f"Invalid shapes: circles = {circles.shape}, expected {(NUM_CIRCLES,3)}"
+        )
+
+    validate_packing_radii(circles[:, -1])
+    validate_packing_overlap_wtol(circles, TOL)
+    validate_packing_inside_rect_wtol(circles, TOL)
+
+    radii_sum = np.sum(circles[:, -1])
+    with open(results_path, "w") as f:
+        json.dump(
+            {
+                "radii_sum": float(radii_sum),
+                "benchmark_ratio": float(radii_sum / BENCHMARK),
+                "eval_time": float(eval_time),
+            },
+            f,
+            indent=4,
+        )
+
+
+if __name__ == "__main__":
+    program_path = sys.argv[1]
+    results_path = sys.argv[2]
+
+    evaluate(program_path, results_path)

problems/alphaevolve_math_problems/circle_packing_rect/input/src/init_program.py

@@ -0,0 +1,22 @@
+# EVOLVE-BLOCK-START
+import numpy as np
+
+
+def circle_packing21() -> np.ndarray:
+    """
+    Places 21 non-overlapping circles inside a rectangle of perimeter 4 in order to maximize the sum of their radii.
+
+    Returns:
+        circles: np.array of shape (21,3), where the i-th row (x,y,r) stores the (x,y) coordinates of the i-th circle of radius r.
+    """
+    n = 21
+    circles = np.zeros((n, 3))
+
+    return circles
+
+
+# EVOLVE-BLOCK-END
+
+if __name__ == "__main__":
+    circles = circle_packing21()
+    print(f"Radii sum: {np.sum(circles[:,-1])}")