DEBS 2026: Call for Challenges

Since 2011, the DEBS Grand Challenge has anchored the community’s benchmarking culture with yearly, data-backed problems. The Challenge has spanned multiple domains, social games, manufacturing equipment, soccer monitoring, smart homes, taxi trips, social networks, RDF, maritime transportation with ML, LiDAR, NILM, AQI, market data, and HDD telemetry, while holding evaluation steady on correctness, throughput, and latency. Some challenges (e.g., the 2015 taxi trips) became de facto benchmarks, pushing stream processing beyond the DEBS limits. This year, we propose an alternative format: a “Call for Challenges” that invites companies to contribute well-specified, evaluable problems backed by representative datasets or secure access.

Why should a company participate? The pitch for your boss/CEO

• Strategic R&D leverage: turn a real backlog problem into a public benchmark; obtain diverse solution ideas without contracting multiple vendors.
• Technical hiring magnet: attract and identify top candidates through leaderboard results, artefact reviews, and finalist Q&A.
• Technology benchmarking: receive an independent, apples-to-apples comparison of relevant methods on your data characteristics (skew, bursts, late events, drift).
• Standards influence: shape evaluation metrics and harness features used by academia and industry in subsequent years.
• Brand amplification: visible “DEBS Challenge Partner” status; invited talk; long-tail citations when the challenge becomes a community benchmark.

Company effort (realistic commitment)

• 1–2 PMs/engineers for 4–6 weeks to package data + define KPIs + author tests.
• Legal review of license/DUA (1–2 cycles).
• One maintainer during the challenge window for Q&A and minor patches.
• Optional on-site presence for finals and award session.

What is needed (submission package)

• Problem statement: domain, queries/tasks, KPIs, constraints, success criteria.
• Dataset and its spec: schema, size, velocity, skew, nulls, anomalies,
  • Operations: access modality (files, API, enclave), reference hardware profile, allowed dependencies.
• Evaluation Infrastructure:
  • Minimum: correctness tests, metric definitions.
  • Good: compute to run experiments
  • Ideal
    • Baseline: naive solution or expected magnitudes.
    • Ground truth: how to test the accuracy of the results
• Legal: data license/DUA, IP expectations, publicity terms, export restrictions.

Intended date

• Lisbon 2026: only for already prepared packages with legal clearance and a complete dry run.
• 2027 (TBA): recommended for new challenges to allow full intake, legal screening, and dry runs.

Timeline (challenge execution, relative)

• T0: Call publication.
• T0+4 weeks: Intent-to-submit (at https://forms.gle/tXSavwNv4uVh7dSK7)
• T0+8 weeks: Full proposals due.
• T0+10 weeks: Technical review + harness dry-run on synthetic slice.
• T0+12 weeks: Notifications; MoUs/DUAs executed.

How is it done (governance and publication)

• Proposals evaluated by the DEBS Grand Challenge Committee.
• Accepted challenges are published on the YEAR.debs.org site with starter kit, rules, and metrics.
• For Lisbon 2026, the proposer must attend, support the evaluation, and confirm results within the proceedings deadlines.
• A challenge paper (encompassing all accepted challenges) is co-authored by the organisers.
• Every participant may submit a short paper to DEBS describing their approach (peer-reviewed).

Evaluation (fixed axes + optional)

• Correctness: per-spec outputs; oracle tests; deterministic replays.
• Throughput: sustained events/sec at ≥X correctness threshold.
• Latency: p50/p95/p99 end-to-end; cold-start overhead; stability under backpressure.
• Optional secondary: memory footprint, cost proxy, energy proxy.

Licensing/IP defaults

• Code: Apache-2.0 for baselines and harness.
• Public datasets: CC BY-NC 4.0 (or ODC-By for tabular) with a no re-identification clause.
• Branding: mutual logo use with pre-approved copy; “DEBS Challenge Partner” designation.

Sponsorship/Prizes [optional]

• In-kind or monetary prizes (hardware credits, cloud credits, cash).
• Awards:
  • Overall Winner (spec-compliant best composite)
  • Best Throughput
  • Best Latency
  • Stability Award
  • Industry Impact
• Benefits to participants: visibility on the leaderboard, talk slots for top teams, artifact badges.