PERSONAL CANON PC02 WEB RESEARCH NOTES

Research date: 2026-08-09

Primary sources consulted

Attention Is All You Need — arXiv

https://arxiv.org/abs/1706.03762

Observed:

• initial submission June 12, 2017;

• multiple later arXiv versions through 2023;

• current abstract reports 28.4 BLEU En-De, 41.8 BLEU En-Fr;

• architecture removes recurrence/convolution from sequence layers;

• paper's Table 1: full self-attention O(n^2 d) complexity, O(1) sequential ops, O(1) max path length;

• recurrent layer: O(n d^2), O(n) sequential ops, O(n) path length;

• paper explicitly says self-attention faster than recurrence when n < d in the stated comparison regime;

• paper notes long-sequence computational performance as a problem and suggests restricted attention;

• parsing experiment extends beyond translation.

NeurIPS proceedings / reviews

https://papers.nips.cc/paper/7181-attention-is-all-you-need https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Reviews.html

Observed:

• official proceedings abstract available;

• official review record includes one reviewer reporting substantial reimplementation work;

• review flags hyperparameter extraction, learning-rate schedule sensitivity and inference differences;

• another review asks for direct long-sentence validation and significance tests.

FlashAttention

https://arxiv.org/abs/2205.14135

Observed:

• later systems paper characterizes Transformers as slow/memory-hungry on long sequences because of quadratic self-attention;

• introduces IO-aware exact attention;

• argues hardware-memory reads/writes are a missing performance dimension;

• shows wall-clock gains while preserving exact attention semantics.

Design conclusion

PC-02 should separate:

original paper claim
author motivation
contemporaneous reviewer concern
later public evidence
private researcher experiment
current researcher belief

The demo should never silently promote later field history into the original paper's evidence.