Systems Track Overview

In addition to the general research track, DASHSys features a dedicated Real-World Systems Track, a competition where participants build production-grade systems for data-centric agents. Submissions are evaluated on the correctness of responses and efficiency of the system.

Winners of the competition will receive monetary prizes.

The DASHSys Systems Track focuses on practical systems that support data-centric agents operating over structured databases and APIs. Participants design agentic systems that answer natural language questions by iteratively executing SQL queries and API calls.

The track emphasizes robust prompt design and system architectures that generalize across models and agentic harnesses. Submissions should demonstrate practical solutions for building reliable agent-driven data systems.

Task Description

Input: A natural language user query.

Output: A response derived by executing SQL queries against a database and/or making REST API calls, along with the full agent trajectory capturing each step taken.

Tools: The following two tools must be implemented in the agent system:

• execute_sql(sql): executes a SQL query against the database and returns results
• call_api(method, url, params, headers): makes an API call against the provisioned sandbox and returns the response

Sample Dataset

Illustration Examples

data.json — 35 labeled examples, each showing a complete agent trajectory. Use these to understand the expected submission format and validate your system. Each entry has five fields:

{
  "query": "<user query>",
  "trace": [
    {
      "step": 1,
      "action": "sql_query",
      "sql": "<SQL executed>",
      "results": ["<...>"],
      "status": "success"
    },
    {
      "step": 2,
      "action": "api_call",
      "api_call": {
        "method": "GET",
        "url": "<endpoint>",
        "params": {"<key>": "<value>"},
        "result_preview": ["<...>"]
      }
    }
  ],
  "answer":   "<final answer>",           // correctness evaluation
  "gold_sql": "<reference SQL query>",    // correctness evaluation
  "gold_api": ["<reference API call(s)>"] // correctness evaluation
}

answer, gold_sql, and gold_api are used for correctness scoring; trace is used for efficiency scoring — see the Evaluation section for details.

Knowledge Graph Database

DBSnapshot — a self-contained set of 18 parquet files representing entities and relationships in the sandbox.

• Dimension tables (dim_*) — one row per entity
• Bridge tables (hkg_br_*) — relationships between entities

Querying the Dataset

There are two ways to retrieve answers for a given query:

import os
import duckdb

SNAPSHOT_DIR = "DBSnapshot"  # path to the KG snapshot folder

# Register every parquet file as a queryable view
con = duckdb.connect()
for fname in os.listdir(SNAPSHOT_DIR):
    if fname.endswith(".parquet"):
        table = fname[: -len(".parquet")]
        con.execute(
            f"CREATE VIEW {table} AS SELECT * FROM read_parquet('{SNAPSHOT_DIR}/{fname}')"
        )

# Run any SQL query against the registered tables
sql = "SELECT * FROM dim_campaign LIMIT 10"
print(con.execute(sql).df())

import requests

# Credentials provided upon registration
CLIENT_ID     = "provided upon registration"
CLIENT_SECRET = "provided upon registration"
IMS_ORG       = "provided upon registration"
SANDBOX       = "provided upon registration"

BASE_URL      = "https://platform.adobe.io"
IMS_TOKEN_URL = "https://ims-na1.adobelogin.com/ims/token/v3"

# Generate bearer token
resp = requests.post(
    IMS_TOKEN_URL,
    headers={"Content-Type": "application/x-www-form-urlencoded"},
    data={
        "grant_type":    "client_credentials",
        "client_id":     CLIENT_ID,
        "client_secret": CLIENT_SECRET,
        "scope":         "openid,AdobeID,read_organizations,additional_info.projectedProductContext,session",
    },
)
access_token = resp.json()["access_token"]

headers = {
    "Authorization": f"Bearer {access_token}",
    "x-api-key":        CLIENT_ID,
    "x-gw-ims-org-id":  IMS_ORG,
    "x-sandbox-name":   SANDBOX,
    "Content-Type":     "application/json",
}

# Make API calls against the provisioned sandbox
response = requests.get(f"{BASE_URL}/<endpoint>", headers=headers)
print(response.json())

API Specifications

API specifications (OpenAPI YAML) can be obtained from: Adobe Experience Platform APIs and Adobe Journey Optimizer APIs.

Deliverables

Participants provide the following for each query:

• A metadata JSON containing the query-specific context selected by the participant's system (e.g., relevant schema and API specifications)
• A filled system prompt with the user query and metadata populated
• An agent output trajectory JSON

Additionally, participants submit:

• A system prompt template with placeholders for the user query and metadata
• A zipped archive of the system's source code

The test set will be released approximately 48 hours before the submission deadline. Participants must submit the deliverables as described above within this window.

Evaluation

Organizers will execute each participant's system prompt using the Claude Agent SDK or OpenAI Agents SDK. The agent harness can be run with any LLM model. For each query, the evaluation harness runs the submitted system prompt against a chosen model (not necessarily the latest) and collects the response and a trajectory JSON capturing the full agent execution trace.

Submissions are evaluated and ranked along two dimensions: correctness and efficiency.

Correctness

The correctness of the generated SQLs, API calls, and the final response is measured against the ground truth. Each component is scored independently:

• SQL correctness: accuracy of generated SQL queries
• API correctness: accuracy of generated API calls
• Response correctness: quality and accuracy of the final answer to the user query

Efficiency

Resource usage during agent execution, measured from the trajectory JSON:

• Number of turns in the agent conversation
• Number of tool calls made during execution
• Total tokens consumed (input + output)
• Wall time for end-to-end execution

Organizers will also run the participant's submitted code to measure end-to-end wall clock time, including any pre-processing and context selection steps beyond the agent execution itself.

Participant-submitted trajectory JSONs will be cross-validated against organizer evaluation runs to ensure reproducibility of results.

Registration

To register your team, submit an abstract on the CMT portal with your team name as the paper title and all team members listed as authors. No document upload is required at this stage. The full system paper and source code are due by the submission deadline.

Submit on CMT

Important Dates

Submission Guidelines

System description papers must follow the VLDB formatting guidelines.

• Maximum 4 pages (excluding references)
• Use the VLDB conference formatting template

Submissions should clearly describe:

• System architecture and design principles
• Prompting or training strategies employed
• Key technical challenges and solutions
• Evaluation methodology with benchmark results and ablation studies
• Lessons learned and open challenges

Review Process

• Double-blind anonymous peer review
• Accepted papers appear in VLDB workshop proceedings

Presentation Format

Organizers

• Vineeth Mohan, Adobe Inc.
• Gromit Chan, Adobe Research
• Raghav Addanki, Adobe Research
• Shaddy Garg, Adobe Inc.
• Aman Sehgal, Adobe Inc.

Contact

For questions about the systems track, dataset, or submission process, please visit the workshop website:

https://dashsys-workshop-vldb.github.io/