Coding datasets for LLMs and coding agents

Repository-level coding datasets for training and evaluating LLMs and coding agents: 60+ environments, 50+ languages and 500 verifiable reinforcement learning tasks.

Talk to an expert

Held-out, executable code data for real-codebase evaluation

For research and evaluation teams whose public benchmark scores have stopped predicting how a model performs inside a real codebase.

Held out

Private environments kept out of public code indexes, so evaluation runs without known training overlap.

Executable

Existing test suites (58.8% average coverage) score outputs by running them, not by string match.

Repository-level

Whole codebases with commit graphs and reviewed pull requests (PRs), not isolated snippets.

Licensed

Provenance and license review before delivery; full metadata available under a non-disclosure agreement (NDA).

Request repository metadata

60+

Over 60 private environments

400+

Over 400 repositories

35M+

Over 35 million lines of code

110K+

Over 110,000 pull requests

50+

Over 50 languages

What are coding datasets for LLMs?

Coding datasets for large language models (LLMs) are collections of real source code, with its commit history, pull requests (PRs), and tests, used to train and evaluate models and coding agents on software engineering tasks. Repository-level datasets keep whole codebases intact, so models are trained and tested on cross-file context rather than isolated snippets.

Public coding benchmarks vs held-out repositories

SWE-bench draws its 2,294 tasks from 12 public Python repositories on GitHub (Jimenez et al., 2024). Public code is routinely crawled for pretraining, and Jain et al. (2024) observed sharp score drops for some models on coding problems published after their training cutoff, a signature of contamination.

Public coding benchmarks compared with held-out private repositories across training overlap risk, task source, verification and language coverage.
Dimension	Public benchmarks (e.g., HumanEval, SWE-bench)	Held-out private repositories
Training overlap risk	High: source is publicly crawlable	Low: not published to public indexes
Task source	Curated problems or public GitHub issues	Reviewed PRs from production codebases
Verification	Unit tests (HumanEval: 164 Python problems; Chen et al., 2021)	Existing project test suites; 58.8% average coverage across the 10 environments that report it
Language coverage	Mostly Python	Ruby, JavaScript, TypeScript, PHP, Java, Kotlin, Python, Go, C#, C++
Licensing	Varies by repository	Provenance and license reviewed before delivery

Sources: Chen et al., 2021, Evaluating large language models trained on code, arXiv:2107.03374 · Jimenez et al., 2024, SWE-bench: Can language models resolve real-world GitHub issues?, ICLR 2024 · Jain et al., 2024, LiveCodeBench: Holistic and contamination free evaluation of large language models for code, arXiv:2403.07974

For evaluation design beyond code, see

LLM evaluation benchmarks

Coding datasets for training, evaluation and reinforcement learning

Repository-level coding datasets

Full repositories with commit graphs and review context for repository-scale tasks.

Code repository datasets

Held-out coding benchmarks

Private environments for evaluation signal with no known training overlap.

LLM evaluation benchmarks

Code datasets with executable tests

Test suites that support execution-based scoring of generated patches.

LLM training data

Pull request datasets for coding agents

Reviewed before/after PR pairs with reviewer comments, used as natural agent tasks.

Agentic SWE evaluation

Coding RL tasks and verifiers

500 coding tasks, each paired with an automated verifier for reward signals.

Coding RL environments

Off-the-shelf AI training datasets

Speech, text, image and video data for non-code workloads.

AI training datasets

Talk to an expert

about your evaluation stack.

How the coding corpus is built and verified

Source from production codebases

Each PR is a reviewed before/after pair from 543 private repositories across 133 environments, built by in-house engineering teams — a median of 13 contributors and 6 years of development history per environment.

Apply the qualification threshold

An environment qualifies at 100 or more pull requests. 66 of the 133 environments clear it, covering 406 of the 543 repositories. The threshold tracks a natural break rather than an arbitrary cut: 61 environments have no pull request history at all, 6 sit between 1 and 99, and 66 clear 100.

Verify by execution

Outputs are scored by running the project's own test suite, the execution-based approach used by HumanEval (Chen et al., 2021) and SWE-bench (Jimenez et al., 2024).

Clear provenance and licensing

License and provenance review is completed before delivery. Appen holds SOC 2 Type II attestation and ISO 27001 certification.

Expert review of agent outputs comes from Appen's software engineering (SWE) contributor network. See SWE-driven evaluation workflows.

Qualifying environments by vertical

60+ environments clear the 100-PR threshold, spanning 400+ repositories and 50+ programming languages. Repository identifiers are withheld; datasets are available under NDA.

Qualifying code environments by vertical, with repository, pull request, lines of code and commit counts.
Vertical	Environments	Repositories	PRs	Lines of code	Commits
Business operations	14	25	31,874	5.6M	111,641
Commerce & marketplaces	9	26	14,072	2.1M	53,504
Sales & marketing tech	12	23	13,921	4.5M	66,108
Logistics & field services	6	27	12,678	2.2M	48,037
HR & workforce	7	107	11,013	12.5M	83,577
Developer & infrastructure tools	6	176	10,695	7.2M	53,798
Consumer & media	4	6	8,778	1.3M	25,557
Healthcare & life sciences	6	12	6,933	1.2M	38,308
Real estate & construction	2	4	547	495K	6,436
Total	66	406	110,511	37.1M	486,966

Most environments are polyglot, so language is reported across the set rather than per row.

Browse code repository datasets

FAQ

Frequently asked questions

What are coding datasets used for?

Coding datasets are used for three things: pretraining and fine-tuning models on realistic software engineering patterns, evaluating code generation against executable tests, and training coding agents with RL. Repository-level data supports multi-file tasks, such as resolving an issue across modules, that snippet-level datasets can't represent.

How do held-out coding benchmarks reduce contamination?

A held-out benchmark uses code that has not been published to public indexes, so it is unlikely to appear in a model's pretraining data. Scores then measure generalization rather than recall. Contamination checks against public benchmark sets add a second control before an environment is released for evaluation.

What is the difference between repository-level and snippet-level coding data?

Snippet-level data pairs a short prompt with a single function, as in HumanEval. Repository-level data keeps the full codebase, commit history, and tests, so a task can require reading several files, editing across modules, and passing the project's own test suite, which is closer to how engineers and coding agents actually work.

Can coding datasets be used for reinforcement learning?

Yes. Coding tasks suit RL because a test suite or verifier can automatically confirm whether an output is correct, producing a reward without human scoring. This approach is called reinforcement learning with verifiable rewards (RLVR). The 500 coding tasks with verifiers are available off the shelf, and custom environments can be scoped separately.

How can we review the data before licensing?

Request repository metadata under NDA. It covers vertical, language, repository count, PR volume, lines of code, and commit counts for each qualifying environment. Teams can then review a sample and the license terms before committing, and confirm that test coverage and language mix match their evaluation or training plan.