Isingformer: Augmenting parallel tempering with learned proposals

The question

Local Monte Carlo moves can struggle to traverse a rugged energy landscape. IsingFormer learns configurations at different temperatures and supplies global proposals to a parallel-tempering search. The paper first tests what the model learns, then studies spin glasses, reusable factorization encodings, and conditional scaling results.

Figure 1

Combine a learned generator with a search process.

A temperature-conditioned Transformer learns from long-run MCMC configurations. It proposes complete configurations to selected replicas of a parallel-tempering ladder, where local updates and replica exchanges continue to propagate promising states.

Left: learning configurations. Long-run MCMC provides configurations at several inverse temperatures. A decoder-only Transformer learns a temperature-conditioned distribution over complete spin states, using both position information and a learned temperature embedding.

Right: using proposals. Generated configurations enter hot and intermediate replicas of a parallel-tempering ladder. Energy evaluation filters proposals, while local updates and replica swaps continue the search. The coldest replicas are not directly supplied with generated proposals.

Figure 1: Combine a learned generator with a search process. — From the paper, p. 2.

Figure 2

Test temperature interpolation and context.

The two-dimensional Ising model offers physical quantities against which to check generated samples. Free energy, magnetization, and clamped-boundary examples test both interpolation to unseen temperatures and conditional generation outside the training setup.

Panels (a) and (b). Free energy tests interpolation between trained temperatures, while magnetization checks the finite-size transition. The comparison uses an exact free-energy reference and Gibbs samples, rather than relying on visual similarity alone.

Panel (c). The half-clamped boundary is absent from training. Energy variance after a single Gibbs sweep tests whether Transformer initialization helps under that changed condition.

Figure 2: Test temperature interpolation and context. — From the paper, p. 4.

Figure 3

Introduce global moves into a spin-glass search.

For three-dimensional spin-glass instances, the plots track residual energy against Monte Carlo sweeps. Turning on proposal moves changes the search trajectory relative to ordinary parallel tempering, with learned and MCMC-generated proposals compared separately.

Panels (a) and (b). The proposal source changes from IsingFormer to long-run MCMC. The dashed line marks activation after 100 sweeps, allowing the subsequent energy reduction to be compared with ordinary parallel tempering under each proposal source.

Figure 3: Introduce global moves into a spin-glass search. — From the paper, p. 6.

Figure 4

Reuse one generator across factorization targets.

An invertible multiplier circuit encodes candidate factors and their product. Training with unclamped product spins creates a reusable generator; clamping a particular product later defines the search task for that target.

Panels (a) and (b). Clamping the product of an invertible multiplier turns factorization into an Ising search problem. Training uses an unclamped universal circuit; at inference, a target product conditions the search while the trained generator is reused.

Panels (c) and (d). The timing comparison covers 440 semiprime instances at a target success probability of 0.99. The example trajectory follows the relative product error for 1,022,117 = 1009 × 1013. The timing is conditional on an already-generated proposal bank.

Figure 4: Reuse one generator across factorization targets. — From the paper, p. 7.

Figure 5

Separate search timing from proposal generation.

The scaling experiment measures time to solution conditional on pre-generated proposal banks. The fitted exponents compare the tested search procedures; generating those proposal banks is excluded, so this is not a claim of an end-to-end speedup including model training and generation.

Reading the axes. The horizontal axis is the maximum product bit length; the vertical axis is the median time to reach a 0.99 success probability. Markers summarize instances, error bars give bootstrap confidence intervals, and dashed lines show exponential fits.

What is timed. This experiment uses proposals generated by long-run MCMC, rather than IsingFormer. Optimization operations are included, but offline proposal-bank generation is excluded. The fitted exponents, 0.846 for PT and 0.564 for TAPT, describe this conditional comparison.

Figure 5: Separate search timing from proposal generation. — From the paper, p. 8.

Supplementary figures 7 figures

Figure S1

Use learned initialization under unfamiliar boundaries.

Spin configurations compare Transformer and random initialization when part of the lattice is clamped. The test probes whether learned context helps the subsequent Monte Carlo evolution.

Panels (a)–(c). The long-run reference is compared with one sweep after Transformer initialization and one sweep from random initialization. Similar-looking configurations suggest a useful starting state, but do not by themselves establish convergence.

Figure S1: Use learned initialization under unfamiliar boundaries. — From the paper, p. 15.

Figure S2

Check the acceptance rule in a controlled circuit.

A small full-adder system makes the stationary distribution directly inspectable. The example shows why energy-only acceptance of external proposals does not generally preserve the Boltzmann target.

Panels (a) and (b). A five-spin full-adder and a controlled three-replica ladder make the target distribution exactly inspectable. Proposals are supplied to two replicas, leaving the coldest replica unaugmented.

Panel (c). The nonzero limiting KL divergence under the energy-only acceptance rule exposes its sampling bias. Exact transition-matrix results and Monte Carlo estimates distinguish optimization usefulness from unbiased equilibrium sampling.

Figure S2: Check the acceptance rule in a controlled circuit. — From the paper, p. 17.

Figure S3

Turn proposal moves off.

The warm-start experiment enables proposals only for the first part of the run. The later trajectories test whether an early advantage persists when the search returns to ordinary parallel tempering.

Panels (a) and (b). Both proposal sources are switched off after the initial 100 sweeps. The later approach toward the PT baseline tests whether the early advantage survives without continued proposal moves.

Figure S3: Turn proposal moves off. — From the paper, p. 18.

Figure S4

Vary proposal coverage and context.

Ablations change which replicas receive proposals and how their context is chosen. These experiments isolate design choices in the proposal mechanism using MCMC-generated configurations.

Panels (a) and (b). The first ablation changes which replicas receive proposals; the second changes how much of the current state conditions a proposal. Both use MCMC-generated candidates, isolating these choices from Transformer training.

Figure S4: Vary proposal coverage and context. — From the paper, p. 19.

Figure S5

Measure proposal acceptance across temperatures.

Acceptance varies across the parallel-tempering ladder. The per-replica plot shows where global candidates are accepted and where the existing configurations are harder to replace.

Reading the replica index. Indices run from hotter to colder replicas. The plot covers proposal-enabled replicas 1–20; replicas 21 and 22 receive no proposals, so there is no acceptance-rate measurement for them here.

Figure S5: Measure proposal acceptance across temperatures. — From the paper, p. 20.

Figure S6

Inspect communication between neighboring replicas.

Swap-acceptance probabilities compare ordinary and augmented parallel tempering. This checks how the added proposal moves affect the exchange pathway through the temperature ladder.

What is compared. The curves measure exchanges between adjacent replicas, combining the alternating swap stages. Their broad similarity checks whether adding proposals substantially changes this communication pathway in the tested ladder.

Figure S6: Inspect communication between neighboring replicas. — From the paper, p. 20.

Figure S7

Use a small logic gate as a distribution check.

The three-spin AND-gate model provides an exactly enumerable example. Its truth table and Boltzmann probabilities connect the circuit encoding to the intended target distribution.

Panels (a)–(c). The Ising graph, truth table, and eight-state distribution connect the encoding to the logical constraint. States 0, 2, 4, and 7 satisfy the AND relation and receive the largest probabilities.

Figure S7: Use a small logic gate as a distribution check. — From the paper, p. 21.

Putting it together

Learned moves, carefully interpreted.

The generator and Monte Carlo search play complementary roles. The energy-only proposal rule is used here as an optimization heuristic; the supplementary analysis explains why unbiased equilibrium sampling requires a fuller correction.

Citation

Bunaiyan, S., Delacour, C., Chowdhury, S., Lee, K., Abdelrahman, A. S., & Camsari, K. Y. Isingformer: Augmenting parallel tempering with learned proposals. Under review (2026). arXiv:2509.23043.

BibTeX

@misc{bunaiyan2026isingformer,
  title = {Isingformer: Augmenting parallel tempering with learned proposals},
  author = {Bunaiyan, Saleh and Delacour, Corentin and Chowdhury, Shuvro and Lee, Kyle and Abdelrahman, Abdelrahman S. and Camsari, Kerem Y.},
  year = {2026},
  eprint = {2509.23043},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  url = {https://arxiv.org/abs/2509.23043}
}