RL, Byte-Level & Long Video

Google

A²RD: Agentic Autoregressive Diffusion for Long Video Consistency

Overview

• Three Google authors (Yale Song, Tomas Pfister, Long T. Le) with two NUS collaborators (Do Xuan Long, Min-Yen Kan); training-free architecture for 1–10 minute video synthesis.
• Frames the long-video problem as semantic drift (entities and environments mutate over time) and narrative collapse (the story stalls or repeats), and decouples creative synthesis from consistency enforcement rather than trying to bake both into a single denoiser.
• Ships LVBench-C, a stress-test benchmark with non-linear entity and environment transitions designed to break long-horizon consistency.

Methodology

• Closed-loop Retrieve–Synthesize–Refine–Update cycle at the segment granularity. Each segment is produced and audited before the next is rolled out; errors do not propagate freely across the full timeline.
• Three named components, each given an acronym in the paper:
  • Multimodal Video Memory (MVM) — a cross-modal store that tracks scene state, characters, environments, and prior narrative beats. Retrieved into the synthesis prompt at every segment so the model has explicit, structured continuity context rather than relying solely on a fixed-length latent window.
  • Adaptive Segment Generation (ASG) — switches among generation modes per segment to balance natural narrative progression with visual consistency. The active mode is selected from the MVM state rather than from a single fixed prompt template, which is how the system avoids the trade-off where a strict consistency prior freezes the narrative.
  • Hierarchical Test-Time Self-Improvement (HTTSI) — refines each segment at both the frame level and the segment level after initial synthesis, using model-as-judge style audits. This is the layer that does the actual error correction inside the loop.
• Mechanism — what A²RD’s closed-loop design actually does:
  • Problem. Pure autoregressive video diffusion drifts because every segment is conditioned only on a short latent context; consistency losses force a small action space and the story stalls. End-to-end long-video diffusion blows up memory.
  • Mechanism. A²RD synthesizes a segment, audits it against the MVM (entities, environments, motifs), refines frames and the segment with HTTSI, then writes new state back to the MVM before generating the next segment. ASG picks the generation mode (continuation vs. transition vs. callback) from the MVM state, so consistency is enforced retroactively in HTTSI rather than proactively as a prior on the denoiser.
  • Why. Decoupling synthesis from consistency means the denoiser stays expressive; consistency is purchased at audit time, not at sample time. Beats prior approaches such as Rolling Forcing and CausVid that compress the entire context into a single forward pass — they trade either consistency or runtime, while A²RD spends a fixed budget per segment regardless of timeline length.

Evaluation & results

• Spans 1- to 10-minute generation across public long-video benchmarks and the new LVBench-C.
• Reports up to +30% on consistency and +20% on narrative coherence against state-of-the-art baselines.
• Human evaluations corroborate the consistency/coherence gains and additionally flag improvements in motion smoothness and transition smoothness — categories the automatic metrics under-weight.
• LVBench-C is specifically designed around cyclical entity/environment appearances with optional state evolutions — a sharper test for long-horizon consistency than benchmarks built on linear narratives.

Ablations

• Paper’s ablations break out the contribution of each pillar (MVM, ASG, HTTSI) separately; removing HTTSI hits frame-level visual consistency hardest, while removing MVM degrades narrative coherence faster than visual quality — which matches the design intent of separating those two failure modes.

Availability

• Project page: dxlong2000.github.io/AARD. Code repo: github.com/dxlong2000/AARD.

Meta

Fast Byte Latent Transformer

Overview

• Meta FAIR (Pagnoni, Ghosh, Zettlemoyer, Han, Iyer) with Stanford (Kallini, Potts) and University of Washington (Limisiewicz, Zettlemoyer); follow-up to BLT (Pagnoni et al., 2024) that fixes its inference cost while keeping the no-tokenizer property.
• Three independent fixes are introduced — BLT-D (diffusion-trained decoder), BLT-S (self-speculative decoding), BLT-DV (diffusion + verification) — none of which require giving up byte-level granularity.
• BLT’s core trade-off restated: byte models eliminate subword tokenization fragility (multilingual, code, numerics, noise robustness) but the local decoder still emits one byte per step, so inference is memory-bandwidth-bound by repeated weight loads.

Methodology

• BLT-D (BLT Diffusion) — the core contribution.
  • Problem. BLT’s local decoder generates one byte per decoder forward pass; even with hierarchical patching, this means many decoder calls per output token equivalent, and on modern GPUs each call pays full memory-bandwidth cost for weight + KV-cache loads.
  • Mechanism. Train the local decoder with two losses at once. The first is the standard next-byte autoregressive cross-entropy on a clean sequence. The second is a block-wise discrete-diffusion loss on a parallel corrupted copy: split bytes into fixed-length blocks (B ∈ {4, 8, 16}), sample a continuous t ~ U(0,1) per training example, and independently mask each byte in the block with probability t. The decoder learns to recover masked bytes given partial context. At inference, BLT-D initializes a block of [MASK] positions and iteratively unmasks multiple positions per decoder step using either confidence-based unmasking (threshold α on per-position probability) or entropy-bounded sampling (largest subset whose cumulative entropy stays below γ). The expensive encoder and global Transformer fire once per block instead of once per patch.
  • Why. Mechanistically similar to BERT-style masked LM applied at byte level inside BLT’s hierarchy, but the auxiliary diffusion loss is co-trained with next-byte prediction so the same weights also support fully autoregressive likelihoods — used by BLT-DV verification and by all likelihood-style evals. Beats pure parallel-decoding approaches that need a separate draft model and separate training.
• BLT-S (BLT Self-Speculation).
  • Problem. Standard speculative decoding requires training a separate draft model with matching tokenizer/vocabulary — not free, and adds deployment complexity.
  • Mechanism. Repurpose BLT’s own local decoder as the drafter. Normally the decoder halts at the next entropy-determined patch boundary; BLT-S instead lets it autoregressively generate up to k ∈ {8, 16} bytes past that boundary, conditioned on the last available latent. The full encoder + global + decoder stack then verifies the k-byte draft in a single forward pass and accepts bytes up to the first mismatch.
  • Why. Under greedy decoding, output is provably identical to standard autoregressive BLT — no quality loss. The cost shifts from many small global model passes to a single big global model pass per accepted draft, which is what the memory-bandwidth math rewards. No additional training, no extra weights, no architectural change.
• BLT-DV (Diffusion + Verification).
  • Problem. One-step block diffusion is the fastest BLT-D variant but generation quality degrades sharply with fewer diffusion steps.
  • Mechanism. Because BLT-D’s weights also support autoregressive decoding (it was co-trained on next-byte prediction), the same model can draft a block via diffusion, then verify the draft with a single causal-mask forward pass, accepting bytes up to the first mismatch. No new model, no new training.
  • Why. Recovers the quality lost in aggressive one-step diffusion by reusing the model itself as the verifier — the verification step bounds drift the way speculative decoding bounds draft errors.
• All variants use the BLT-1T pretraining mixture (1T tokens, public sources including a Datacomp-LM subset). 1B-param models trained for 240k steps, 3B-param models for 480k steps. Efficiency reported as decoder NFEs, encoder/global NFEs, and an estimated 16-bit memory-bandwidth figure — explicitly proxy metrics, not wall-clock.

Evaluation & results

• Generation tasks: French→English and German→English translation on FLORES-101 (4-shot, SentencePiece BLEU); HumanEval (0-shot, pass@1); MBPP (3-shot, pass@1).
• Likelihood benchmarks: ARC-Easy, ARC-Challenge, PIQA, HellaSwag, MMLU. BLT-D scores approach BLT’s baseline on all five, confirming the diffusion objective does not damage autoregressive reasoning.
• Reported memory-bandwidth reductions at 3B parameters:

  Variant	Estimated memory-bandwidth reduction vs. BLT	Quality posture
  BLT-D-4 (block 4)	~50% (over half)	Nearly matches BLT on tasks
  BLT-D-16 (block 16)	87–92% (fastest configuration)	Score drops on HumanEval / MBPP
  BLT-S (k=16)	Up to 77%	Identical to BLT under greedy decoding
  BLT-DV (1-step diffusion + verify)	Up to 81%	Recovers quality lost by 1-step diffusion

• Translation tasks benefit most across all block sizes; coding tasks (HumanEval, MBPP) are the most block-size-sensitive, with BLT-D-16 trading meaningful pass@1 for max throughput.

Ablations

• Generation-diversity ablation: using entropy-bounded sampling with top-p, more decoder NFEs correlate positively with type-token ratio. The efficiency–diversity trade-off is therefore tunable at inference time via α and γ without retraining.
• The paper is explicit that NFE-based memory-bandwidth numbers are proxy metrics; wall-clock improvement requires an optimized kernel implementation, flagged as the most important follow-up.

Availability

• Authors: Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz, Gargi Ghosh, Luke Zettlemoyer, Christopher Potts, Xiaochuang Han, Srinivasan Iyer.
• Companion repo for prior BLT: github.com/facebookresearch/blt.

Tencent (Hunyuan)

Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex

Overview

• Tencent Hunyuan team (Yun Qu, Qi Wang, Yixiu Mao, Heming Zou, Yuhang Jiang, Yingyue Li, Wutong Xu, Lizhou Cai, Weijie Liu, Clive Bai, Kai Yang, Yangkun Chen, Saiyong Yang, Xiangyang Ji).
• #1 paper of the day on Hugging Face Daily Papers, May 11.
• Argument: existing group-based RLVR methods (GRPO, RLOO, DAPO, GSPO, and variants) implicitly define a target distribution on a response simplex and step toward it by first-order approximation — but never write that target down. LPO writes the target down and minimizes a divergence to it explicitly.

Methodology

• Mechanism — Listwise Policy Optimization (LPO):
  • Problem. Group-based RLVR samples a group of responses per prompt and updates the policy via group-relative advantage signals. Every recipe in this family (GRPO, RLOO, DAPO, GSPO) ends up with a slightly different update rule, and it has not been clear what they all share or what individual choices buy you. Empirically the rules differ on training stability and on response diversity — the modes that get suppressed depend on which advantage normalization you picked.
  • Mechanism. Restrict the proximal RL objective from the full action space to the response simplex spanned by the sampled group. On that simplex, each existing group-based method is reinterpreted as projecting the policy toward an implicit target distribution via a first-order Taylor expansion of the proximal objective. LPO instead makes the target distribution explicit and projects the policy onto it via exact divergence minimization — the divergence is a hyperparameter and can be KL, reverse KL, χ², or any well-behaved choice, each with different structural properties. The implicit-target view is the unifying lens; explicit divergence minimization is the new algorithm.
  • Why. Because the target is now explicit, LPO inherits two properties the implicit methods do not provably have: monotonic improvement on the listwise objective and bounded, zero-sum, self-correcting projection gradients. Bounded gradients explain why training is more stable; zero-sum normalization explains why response diversity is preserved (rewards do not collapse onto a single mode); self-correcting projection means a bad early step gets undone in the next projection rather than compounding. Beats GRPO and family on matched targets without changing the reward model or the sampling budget — the gain is purely from making the implicit target concrete and minimizing the right divergence.
• Framework is divergence-agnostic: each choice of divergence yields a distinct member of the LPO family with different bias/variance characteristics. This is the axis the paper trades off against the implicit-target methods on.

Evaluation & results

• Evaluated across diverse reasoning tasks and multiple LLM backbones (specific model families enumerated in the paper).
• LPO consistently improves training performance over typical policy-gradient baselines under matched targets — i.e., when the implicit target of each baseline is reproduced as LPO’s explicit target. The improvement is attributable to the projection step, not to a different reward signal or different sampling.
• Optimization stability preserved: bounded zero-sum projection gradients suppress the training spikes that GRPO and DAPO trade off against on long-horizon tasks.
• Response diversity preserved: the listwise projection does not collapse the policy onto a single high-reward mode the way some advantage-normalization schemes do empirically.

Ablations

• Ablations cover divergence choice — distinct divergences yield distinct structural properties on the projection step (paper reports relative differences on stability vs. diversity vs. fit).
• Matched-target ablation isolates LPO’s benefit from the choice of advantage signal: under the same target distribution, exact divergence minimization beats first-order approximation across backbones.