CWL Scatter and Cross-Product
CWL scatter in both forms: a single-parameter scatter over an array input, and a two-parameter scatter with scatterMethod flat_crossproduct. The scatter dimensionality is what distinguishes correct CWL engines from approximate ones.
cwlVersion: v1.2
class: Workflow
id: scatter-shards
label: Scatter over shards
doc: Inert fixture exercising CWL scatter and its scatterMethod variants.
requirements:
ScatterFeatureRequirement: {}
MultipleInputFeatureRequirement: {}
StepInputExpressionRequirement: {}
inputs:
shards:
type: File[]
regions:
type: string[]
default: [north, south]
outputs:
counts:
type: File[]
outputSource: count_shard/record_count
steps:
count_shard:
run: cwl-command-line-tool.cwl
scatter: extract
in:
extract: shards
out: [record_count]
cross_product:
run: cwl-command-line-tool.cwl
scatter: [extract, skip_header]
scatterMethod: flat_crossproduct
in:
extract: shards
skip_header:
valueFrom: $(true)
out: [record_count]
Specifications
- Cwl Version
- v1.2
- Class
- Workflow
- Scatter Steps
- 2
- Scatter Method
- flat_crossproduct
- Array Inputs
- 2
Testing contract
Expected to pass- Scenario
- Determine how many jobs a CWL scatter step expands to under each scatterMethod
- Expected result
- The single-parameter scatter expands once per shard, while the flat_crossproduct step expands to the product of its two scattered inputs
What is a .cwl file?
CWL (Common Workflow Language) is an open standard for describing command-line analysis workflows portably. A .cwl document is YAML (or JSON) declaring a cwlVersion and a class of CommandLineTool, Workflow, or ExpressionTool, with typed inputs and outputs, input bindings that map values onto command-line arguments, and — for workflows — steps wired together by source references. Requirements and hints declare the container image and resources a step needs.
How to use this file
Use an example .cwl file to test workflow parsers, validators, and execution engines, verifying that step dependencies form the expected graph, that input bindings produce the documented command line, and that an unsupported cwlVersion is rejected.
How to use this file for testing
“CWL Scatter and Cross-Product” is a deterministic Novus Examples fixture for Graph data, Data engineering, Conversion testing. Node/edge datasets in GraphML and GEXF (directed and undirected, with attributes and weights) — for testing network importers, layout tools, and graph converters.
Documented properties for this file: CWL · 808 bytes. Compare results against paired or grouped companions on this page when present (clean↔damaged, searchable↔scanned, or format twins) so scores stay reproducible across runs.
Download the file once, keep the path stable in CI or local scripts, and treat the spec table as the contract: dimensions, seeds, field lists, and roles are intentional. Corrupt or invalid samples are labelled as such — expect parsers to fail loudly rather than silently accept them.
Pipeline and infrastructure fixtures are inert configuration: steps reference fictional images and scripts, and nothing here executes. Run your linter, schema validator, migrator, or policy engine against them, and expect the deprecated-syntax and intentionally invalid variants to be rejected.
Related files
- cwlCWL Workflow (Six-Step Topology)A CWL Workflow carrying the same six-step ETL topology as the Airflow and Argo fixtures, with dependencies expressed as step output references and a multi-source input resolved by pickValue. For cross-format DAG conversion tests.
- pyAirflow DAG Definition (Operator Style)An Airflow DAG definition in the classic operator style, using only the no-op EmptyOperator so the file describes a topology and performs no work. Carries the six-task ETL graph shared across the Airflow JSON, Argo, Graphviz and Mermaid fixtures.
- pyAirflow TaskFlow API DAGA decorator-based Airflow TaskFlow DAG where dependencies are implied by function calls rather than >> operators — the shape static DAG extractors most often get wrong. Every task returns a literal, so nothing performs I/O.
- jsonAirflow-Shaped Serialized DAG (JSON)The serialized-DAG shape Airflow stores in its metadata database, as standalone JSON: per-task metadata with explicit downstream_task_ids. Carries the same six-task topology as the Python, Argo, Graphviz and Mermaid fixtures.
- yamlArgo Workflow DAG TemplateAn Argo Workflows DAG template carrying the same six-task ETL topology as the Airflow, Graphviz and Mermaid fixtures in this category, so a converter or visualiser can be scored against one known answer across four formats.
- yamlArgo Workflow Steps TemplateArgo's steps template, whose double-nested list is a genuine parser trap: the outer list is sequential and the inner list is parallel, so a reader that flattens it reports four sequential steps instead of three groups.
Generated by generation/pipelines.py. Free for any use, no attribution required — license.