Index: arXiv AI research velocity

What this measures

The count of distinct papers first submitted to arXiv within a given calendar month whose primary or secondary category is either cs.AI (Artificial Intelligence) or cs.LG (Machine Learning).

Read it as: a raw velocity signal for the AI/ML research surface area — how many new papers are being produced per month.

What this does not measure

• Paper quality, novelty, or reproducibility
• Citation impact
• Whether research is original or derivative
• Industrial vs. academic output
• Geographic distribution
• Duplicate, withdrawn, or replaced papers (v0.1 — see Known limitations)

If you want any of those, combine this index with other sources. On its own it is a volumetric indicator, nothing more.

Exact query

For each calendar month in the trailing 24-month window:

(cat:cs.AI OR cat:cs.LG) AND submittedDate:[YYYYMMDD0000 TO YYYYMMDDHHMM2359]

Evaluated against https://export.arxiv.org/api/query. Count is read from the opensearch:totalResults element of the returned Atom XML response.

Deduplication: arXiv's search backend returns unique entries for OR queries across categories. A paper cross-listed in both cs.AI and cs.LG is counted once per query.

Rate limit: 3.1 seconds between requests (arXiv polite-use guidance).

Pipeline

Source: scripts/build_index_arxiv_ai.py

arXiv API → XML parse → monthly counts → CSV → SVG

Implementation: Python 3.11+ stdlib only. No third-party dependencies. Fully reproducible from a clean Python environment in ~75 seconds.

Outputs

Git history preserves every past state — that is the audit trail for any brief that cited a specific value at publication time.

Output schema

Canonical CSV: data/indices/arxiv-ai-velocity.csv

Header row is exactly month,papers. Rows sorted chronologically ascending. Exactly 24 data rows. UTF-8, LF line endings, no BOM.

Known limitations (v0.1)

1. No resubmission handling. arXiv submittedDate refers to first submission; papers replaced or updated within a month may or may not recount depending on arXiv index behavior.
2. No withdrawal filter. Papers later withdrawn are still counted in the month they were first submitted. This is correct behavior for a velocity signal but worth stating.
3. No seasonality adjustment. The raw series shows conference-deadline clustering (NeurIPS, ICML, ICLR, CVPR). A seasonally adjusted variant is planned for v0.2.
4. Category overlap with non-AI fields. cs.LG includes some papers that are ML-method applied to non-AI domains. Accepted as part of the definition.
5. Single-point-in-time snapshots. arXiv backfills corrections and reclassifications. See ADR-0010.
6. Timezone: arXiv UTC. No adjustment.

Changelog

v0.1 — 2026-04-11 — Initial release. 24-month trailing window, monthly cadence, stdlib-only pipeline, SVG rendering.

Methodology changes bump the version and append here. Historical CSV values are preserved in git so briefs citing older versions remain verifiable.

How to reproduce

git clone https://github.com/notifuturo/cyberfuturo
cd cyberfuturo
python3 scripts/build_index_arxiv_ai.py
# Writes data/indices/arxiv-ai-velocity.{csv,svg}
# Takes ~75 seconds (24 requests at 3.1s rate limit)

Zero third-party packages. No API keys. No environment variables.

Deprecation criteria

Per business rule BR-11, every index has pre-committed conditions for deprecation, renaming, or retirement. This index is deprecated when any one of the following becomes true:

1. Source deprecation — arXiv removes or rate-limits the public /api/query endpoint, or requires authentication that blocks free-tier usage.
2. Source drift beyond tolerance — re-snapshots show >3% drift on month values older than 12 months, indicating backfill behavior that breaks the trailing-window assumption.
3. Category restructuring — arXiv reorganizes the cs.AI / cs.LG hierarchy in a way that makes a single query no longer meaningful.
4. Measurement becomes unrepresentative — if AI/ML research shifts away from arXiv to a successor preprint server.
5. Replaced by a better version — a v2.x index with the same slug and stricter methodology can supersede v0.x.

Deprecation process: publish a final versioned snapshot, announce with explicit reasoning, archive the CSV/SVG, and strike through the index with a link to the final snapshot. The index is never silently removed — every deprecation is logged in this document's changelog.

Contact

Methodology errors, data disputes, or reproducibility problems: open an issue on the repo.