Competition Track · ICSE 2027

Benchmarking VLMs for in-car scene understanding.

The first competition for testing and advancing Vision-Language Models (VLMs) in in-car interior scene understanding. Evaluate systems on seat occupancy, seat-belt status, passengers, objects, and driver behavior across two tracks using a shared feature definition. The competition will be held at ICSE 2027 in Dublin, Ireland.

Compare tracks Get the starter kit ↗

ISU-Test · Paper Dataset · Hugging Face ISU-Test · GitHub

driver

passenger

seat belt

phone

Example interior in-car scene detections.

Why this competition

Cameras see the cabin fine. Understanding it is the hard part.

In-cabin monitoring is increasingly employed for detection of safety-relevant events - such as driver distraction, unbelted passengers — as well as for comfort and personalization. VLMs read those camera images flexibly, but can provide confident, incomplete, or simply wrong descriptions. Real test cases are expensive to collect at scale. The ISU-Challenge targets A) a realistic interior scene dataset based on synthetically generated images to test ISU systems automatically, and B) a robust in-car scene understanding system that is robust to challenging in-car scenes.

Dual-track design

Track A generates the failure-inducing scenes; Track B builds the systems that must survive them. Both share one feature schema, so progress on either track is directly comparable.

Ground-truth preserving

Track A scenes must stay semantically and geometrically faithful to their source — appearance changes only, so existing labels remain valid without re-annotation.

Executable evaluation

Both tracks are scored by automated pipelines against reference VLMs — Qwen/Qwen2.5-VL-3B-Instruct during development, an industrial system and withheld real+synthetic data at final evaluation.

Dataset

Same scene, multiple channels.

The starter kit pairs every synthetic scene with its auxiliary channels; a real cabin photo is shown alongside for the domain Track B is ultimately scored on. Full dataset available on Hugging Face.

Synthetic RGB in-car scene — Rendered scene (RGB)

Instance segmentation map of the same scene — Segmentation map

Canny edge map of the same scene — Canny edge map

Real in-car photograph — Real scene (Track B)

Same synthetic scene across channels, and a real cabin photo used in Track B.

Worked example

Example scene description.

The Track A sample scene above: a male driver, belted; two rear passengers; a phone, cola bottle, and cola can on the codriver seat.

Synthetic scene used in the worked feature-label example

Competition tracks

Two tracks, one cabin.

Teams may enter one track or both. Track A challenges perception systems with generated scenes; Track B is the perception system being tested.

Test generation

Track A · Find failures

Build automated test generators that diversify synthetic base scenes — textures, materials, passenger appearance and clothing — while leaving scene geometry, semantics, and spatial relationships untouched, so the source scene's ground truth stays valid.

Appearance-level transforms only: no adding, removing, moving, or reshaping annotated people or objects.
Any transform that does change scene structure must ship consistent, regenerated ground truth alongside it.
Scored against a reference VLM during development; an industrial system is added at final evaluation.

InputSynthetic base scenes + ground-truth labels + auxiliary channels (depth, segmentation)

OutputTransformed scene dataset + reproducible generation code

Perception robustness

Track B · Survive them

Build ISU systems that answer safety-relevant questions about synthetic and real cabin scenes — occupants, seat belts, objects, driver and passenger behavior — returning predictions in the prescribed JSON schema.

Reference evaluator model: Qwen/Qwen2.5-VL-3B-Instruct.
Trained and validated on synthetic scenes plus 50 collected real cabin photos.
Final ranking is done on a private test set spanning both synthetic and real domains.

InputSynthetic + real cabin scenes, paired with the competition questions

OutputModel, parameters, reproducible inference code, JSON-formatted predictions

Starter kit

Everything you need to start building.

Synthetic base scenes, ground-truth labels, auxiliary channels, and reference notebooks for both tracks — clone it and start building against Track A or Track B today.

track_a/participant_starter.ipynb track_a/track_a_evaluator.ipynb track_b/participant_starter.ipynb track_b/track_b_evaluator.ipynb

Get the starter kit on GitHub →

Evaluation

Automated evaluation.

Both tracks are scored by pipelines that run against the same reference model using default metrics.

Track A metrics

Failure rate

Scenes exposing a failure ÷ scenes executed, via feature matching against Qwen/Qwen2.5-VL-3B-Instruct. A separate industrial system is added for final evaluation.

Realism

Measured with published image-realism metrics [2].

Diversity

Feature diversity of failing scenes, following ISU-Test [1]; visual diversity measured with CLIP (clothing colour, style, and similar appearance shifts).

Efficiency

Generation time, measured on a standardized g6e.2xlarge EC2 instance at final evaluation.

Track B metrics

Accuracy

Feature-level and scene-level exact-match performance on visual-question-answering predictions.

Latency

Time to process a scene and return a prediction, measured on g6e.2xlarge with an NVIDIA L40S GPU.

Extended evaluation

Submitted systems are re-scored on an extended real + synthetic dataset withheld until final evaluation.

Track A results are ranked by Pareto non-dominance sorting across all four objectives; ties fall back to a weighted linear combination.

5,000

synthetic in-car scenes

real cabin photos

scene features

MIT

license on non-external data

Leaderboard

Live rankings, once scoring opens.

Track A and Track B standings will populate here as submissions are scored against the reference pipelines.

To be announced

No submissions have been scored yet — the leaderboard opens alongside the competition launch and starter-kit release.

Timeline

Synchronized with ICSE 2027.

All dates below are synchronized with the ICSE 2027 Competition Track schedule. Learn more about the conference: ICSE 2027.

Sep 2026Competition launch, starter-kit release

27 Nov 2026Participant submissions (data & reports)

4 Dec 2026Competition report submissions

16 Dec 2026Reviewer response issued

20 Jan 2027Camera-ready deadline

25 Apr – 1 May 2027ICSE 2027 in-person sessions, Dublin

Participate

How to enter.

Registration & submission

Email lev.sorokin@tum.de with subject ISU-Challenge Registration Track A/B, stating team or participant name, affiliation, and address — a team name is encouraged if you're entering as a group.

Build against the starter kit for one or both tracks. Registration deadlines and platform details are announced at competition launch. Ongoing communication with organizers and other teams happens over Slack; an invite is sent after registration.

Track A submission: generated image archive plus reproducible generation code, preserving the required semantic and spatial ground truth.

Track B submission: model files, parameters, inference or training code, and predictions in the required JSON format.

Submissions pass automated integrity and reproducibility checks, and all submissions are additionally reviewed manually.

Teams are invited to submit solution papers to the ICSE 2027 Competition Track proceedings. Submissions are reviewed on soundness, writing quality, and replicability. Novel approaches can be invited on methodological merit even outside the top three track score rankings. Authors of accepted papers will be invited to present their approach at the conference in Dublin, Ireland in May 2026. Accepted papers will be published in the conference proceedings.

Organizers

Your organizers.

Lev Sorokin

BMW Group & Technical University of Munich

Rifaath Ameen

BMW Group & FAU Erlangen-Nuremberg

Stefano Carlo Lambertenghi

Technical University of Munich & fortiss GmbH

Chen Yang

Technical University of Munich

FAQ

What you need to know.

A Track A

What counts as a valid test input?

Any synthetic in-car scene from the starter kit, together with its available auxiliary channels — depth maps, semantic segmentation, edge information.

What must the output look like?

A transformed scene dataset plus the reproducible code used to make it. Each scene must preserve the semantic content and spatial geometry of its source, so the provided ground-truth labels stay valid.

What does "preserving ground truth" mean?

Appearance may change — clothing, texture, material — but location and geometry may not. No adding, removing, moving, or reshaping annotated people or objects, and no altered spatial relationships.

Do I need to generate new labels?

No — Track A is built around label-preserving transforms, so the existing labels remain valid as-is.

What if my transform changes the segmentation?

Then the scene no longer satisfies Track A's validity requirement and won't score.

Can I move, add, or remove objects or people?

No — those operations invalidate the provided ground truth and are not permitted.

Can I use LLMs or diffusion models to generate images?

Yes, provided results stay realistic, reproducible, and semantically valid. Declare any model or API dependencies, and respect applicable token or compute limits.

B Track B

What does the system need to predict?

The observable features in the label JSON — phone use, occupancy, seat-belt status, suitcase presence/location, beverage objects, baby-seat configuration, and related cabin features — using the schema's feature names and allowed values.

Where do I get real training data?

50 organizer-collected real cabin images with labels. A recording script and instructions are provided if you want to collect more.

What format should predictions use?

Strict JSON using the prescribed feature names, one value per evaluated feature, chosen from that feature's allowed values.

How is accuracy calculated?

Feature-level comparison against ground truth, plus scene-level exact match when every evaluated feature for a scene is correct. Missing predictions count as incorrect.

Which images are used for evaluation?

Starter-kit synthetic and real images for development; a withheld extended dataset spanning both domains for final evaluation — don't over-fit to one visual domain.

Which model should I use?

The starter kit ships a baseline on Qwen/Qwen2.5-VL-3B-Instruct, but final evaluation runs on whatever system you submit — the baseline isn't mandatory.

How is latency measured?

Time to process one image and return its Track B prediction, on a g6e.2xlarge EC2 instance with an NVIDIA L40S GPU, for comparable numbers across teams.

Can I use external data or models?

Yes — document external datasets, pretrained models, and dependencies, and provide reproducible inference or training code. All artefacts must comply with their licenses.

Citation

How to cite this work.

If you use the ISU-Challenge dataset in your research, please cite the following work:

@inproceedings{sorokin2026isutest,
  author    = {Lev Sorokin and Chen Yang and Ken E. Friedl and Andrea Stocco},
  title     = {Search-based Testing of Vision Language Models for In-Car Scene Understanding},
  booktitle = {Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE 2026), Industry Track},
  year      = {2026},
  doi       = {10.1145/3832783.3834506}
}

Supporters

Supported by industry and academia.

Friedrich-Alexander-Universität Erlangen-Nürnberg