Vector — Log Topology in TOML with Multi-Line Transforms (toml)
A log-processing topology in TOML where components declare their inputs rather than an explicit graph, including a dropped-event route consumed as parse_json.dropped. The transform bodies are triple-quoted multi-line strings holding a different language, which is what breaks TOML tooling here.
# Vector topology: OTLP and file sources, remap transforms, two sinks.
# Components are wired by naming their inputs, so the graph is implicit.
[sources.otlp_logs]
type = "opentelemetry"
grpc.address = "0.0.0.0:4317"
http.address = "0.0.0.0:4318"
[sources.container_logs]
type = "file"
include = ["/var/log/containers/*shop*.log"]
read_from = "end"
fingerprint.strategy = "checksum"
[transforms.parse_cri]
type = "remap"
inputs = ["container_logs"]
source = '''
. |= parse_regex!(.message, r'^(?P<ts>\S+) (?P<stream>stdout|stderr) (?P<tag>[FP]) (?P<body>.*)$')
.timestamp = parse_timestamp!(.ts, format: "%+")
'''
[transforms.parse_json]
type = "remap"
inputs = ["parse_cri", "otlp_logs"]
drop_on_error = false
reroute_dropped = true
source = '''
parsed, err = parse_json(.body)
if err == null {
. = merge(., object!(parsed))
}
.level = downcase(string!(.level || "info"))
'''
[transforms.redact]
type = "remap"
inputs = ["parse_json"]
source = '''
if exists(.customer_id) { .customer_id = sha2(string!(.customer_id), variant: "SHA-256") }
del(.session_id)
del(.device_id)
'''
[sinks.http_logs]
type = "http"
inputs = ["redact"]
uri = "https://logs.example.com/v1/logs"
encoding.codec = "json"
batch.max_events = 1000Specifications
- Sources
- 2
- Transforms
- 3
- Sinks
- 2
- Multiline String Blocks
- 3
- Has Dropped Output Route
- true
- Graph Is Implicit
- true
- Hashes Customer Id
- true
Testing contract
Expected to pass- Scenario
- Parse the TOML and build the component graph from the inputs fields.
- Expected result
- Seven components resolve into a connected graph with no dangling input, the three triple-quoted transform bodies survive intact including their backslashes, and parse_json.dropped resolves to the transform's error output.
What is a .toml file?
TOML (Tom's Obvious Minimal Language) is a configuration format designed to be unambiguous and easy to read, with typed values, sections in bracketed tables, and clear date and number handling. It maps cleanly to hash tables. It is widely used for application and package configuration, such as Rust's Cargo and Python's pyproject.
How to use this file
Use an example TOML file to test config parsers, typed-value and table handling, and tools that read project or application settings.
How to use this file for testing
“Vector — Log Topology in TOML with Multi-Line Transforms (toml)” is a deterministic Novus Examples fixture for Observability, Config parsing, Config testing. Structured and plain-text telemetry with known timestamps, levels, request identifiers, and error states for testing log ingestion, correlation, dashboards, and alert pipelines.
Documented properties for this file: TOML · 1,346 bytes. Compare results against paired or grouped companions on this page when present (clean↔damaged, searchable↔scanned, or format twins) so scores stay reproducible across runs.
Download the file once, keep the path stable in CI or local scripts, and treat the spec table as the contract: dimensions, seeds, field lists, and roles are intentional. Corrupt or invalid samples are labelled as such — expect parsers to fail loudly rather than silently accept them.
Telemetry fixtures use fixed trace IDs, span IDs, and timestamps so ingestion is reproducible run to run. Point your collector, parser, or query layer at the file and assert the documented span tree, metric families, or severity mix; service and host names are invented.
Code examples
import tomllib
with open("vector-topology.toml", "rb") as f:
data = tomllib.load(f)
print(data)Related files
- yamlAlertmanager Config — Routing Tree, Inhibition and Mute Windows (yaml)An Alertmanager configuration with the routing behaviour that is hard to reason about without a fixture: a continue: true branch so a critical alert reaches two receivers, an inhibition rule suppressing warnings under a matching critical, and a weekend mute window.
- yamlOpenTelemetry Collector — Tail-Sampling Policies (yaml)Six tail-sampling policies including an inverted regex match that drops health checks and a composite policy with its own rate limit and ordered sub-policies. Policies are OR-ed and one match keeps the entire trace, which is the semantic most people get backwards.
- yamlOpenTelemetry Collector — Three-Signal Pipeline Configuration (yaml)A Collector configuration wiring traces, metrics and logs through separate pipelines, with named component instances, a filelog receiver that parses the CRI log fixture, OTTL statements performing the semantic-convention migration from the paired trace fixtures, and a redaction processor.
- yamlPrometheus Alerting Rules — Four Rules Across Two Groups (yaml)Four alerting rules over two groups, using the YAML features that break naive config readers: block scalars for multi-line PromQL, folded scalars for descriptions, Go template annotations with pipes, and keep_firing_for. Every expression references metrics that exist in this category's fixtures.
- yamlPrometheus Scrape Config — Relabelling and Federation (yaml)Three scrape jobs with the parts that actually matter: relabel_configs rewriting a target label with a capture group, metric_relabel_configs dropping the two unbounded labels from the cardinality fixture, honor_timestamps for the textfile collector and a federation job with a match[] selector.
- jsonDashboard JSON — Chained Template Variables (json)A dashboard driven by four template variables, one of which queries using the value of another — the chained dependency that must be resolved in order. Multi-select variables interpolate as regex alternations, which is why every query here uses =~ rather than =.
Generated by generation/observability.py. Free for any use, no attribution required — license.