Most observability budgets blow up on logs first. Traces and metrics scale predictably; logs scale with whatever a developer happened to write in a request handler. A single noisy service can double an observability bill in a quarter. Worse, by the time anyone looks at the bill, the noisy logs are entrenched — used in dashboards, referenced in runbooks, depended on for compliance.
The answer is not to disable logging. It is to apply the same engineering discipline to logs that gets applied to data pipelines: classify what is stored, route it appropriately, and drop what is not earning its cost. This guide covers the practical techniques that consistently cut log spend by 50% or more.
Where Cost Actually Goes
A log line in your platform consumes spend in four places:
- Network egress. From the source to the collector, often crossing AZ or region boundaries.
- Ingest processing. Parsing, enrichment, indexing in the backend.
- Storage. Hot, warm, or cold — each tier with different per-GB pricing.
- Query. Per-query or per-GB-scanned, depending on the backend.
The biggest savings usually come from steps 2 and 3. Network egress matters at extreme scale; query cost matters for active investigation but not for baseline spend.
Triage by Log Class
Not all logs are equal. A practical taxonomy:
- Audit logs. Compliance-driven. Must be retained, immutable, and accessible for audit. Volume is moderate, the value per line is high.
- Application logs. What developers
log.infoandlog.error. Volume varies enormously by code quality. - Access logs. Per-request records from load balancers, gateways, and web servers. High volume, low value per line.
- Debug logs. Verbose runtime detail. Useful during incidents, noisy otherwise.
- Metrics-as-logs. Lines that exist only to be aggregated into a metric.
Each class deserves a different cost strategy. Treating them all as one bucket is what drives bills to grow linearly with traffic.
Drop What Should Not Have Been Written
The cheapest log line is the one that was never sent. Start every cost-optimization exercise here.
Common "why is this even being logged" findings:
- Health-check requests at INFO level. Often 90%+ of access log volume.
- Successful requests at DEBUG level. Useful in development; pure cost in production.
- Stack traces logged twice — once where they're caught, once where they're rethrown and caught again.
- Third-party library debug logs left on by default.
- Logs of internal AWS service polling traffic (SQS long polls, S3 list operations).
A week-long audit of the highest-volume log streams in your backend almost always finds a few of these. Removing them at the source — code change or config change — is the cleanest fix.
Sample What Is High-Volume but Low-Value-Per-Line
For log streams that earn their existence but produce way more lines than anyone reads, sampling is the next lever.
Straightforward sampling — keep 10% of records randomly — works for access logs and similar uniform streams. The catch is errors: a flat 10% sample drops 90% of error responses too.
Stratified sampling is more useful:
processors:
filter/keep_errors:
logs:
log_record:
- 'severity_text == "ERROR" or attributes["http.status_code"] >= 500'
probabilistic_sampler/baseline:
sampling_percentage: 5
Keep all error and slow records. Sample everything else aggressively. The error rate in raw data is preserved; the volume drops by 90%+.
Most log routers (Vector, Fluent Bit, OpenTelemetry Collector) support sampling at the pipeline level. Push it as close to the source as possible — sampling at the backend pays for ingest you are about to throw away.
Aggregate Instead of Storing
If a log line exists primarily to be counted, do not store it as a log. Convert it to a metric at the collector.
A common pattern: a service emits an INFO log for every successful API call, partly for visibility and partly so dashboards can graph request rate. The dashboards would work just as well off a counter metric. Drop the log lines, increment a counter at the collector, save 95% of the volume.
Fluent Bit, Vector, and OpenTelemetry's count processor all support this conversion. A few well-chosen aggregations replace large volumes of per-line storage.
Drop Fields You Do Not Need
Within a log line, fields are not all equal. A typical structured log might include the full request URL, all request headers, an environment dump, a hostname, and a process ID. Storing all of those is wasteful when queries only filter on three fields.
Fluent Bit and Vector both support field-level dropping. A configuration that strips noise from access logs:
transforms:
strip_access_logs:
type: remap
inputs: ["access_logs"]
source: |
del(.request_headers)
del(.response_headers)
del(.environment)
del(.process_id)
Field-level drops often cut log storage cost in half on access-heavy services. They are invisible to dashboards built on the remaining fields.
Tier by Age
Most logs are hot for hours and cold for the rest of their retention. The query patterns reflect this — the active incident uses logs from the last hour, audits sample logs from days or weeks back.
Log backends increasingly support tiered storage with different price points:
- Hot tier. Recent logs (24-72 hours), indexed, fast query.
- Warm tier. Days to weeks, slower query, lower cost.
- Cold tier / archive. Object storage, archived for compliance or rare access.
The price difference between hot and cold is typically 5-10x. A pipeline that ages logs through tiers — Elasticsearch ILM policies, Loki object-store backends, Datadog rehydratable archives — captures most of the storage savings without losing access.
The gotcha: query cost on cold tiers is sometimes per-GB-scanned. Querying a year of archived logs to investigate one incident can cost more than a month of hot ingest. Use cold tiers for retention, not exploration.
Watch the Cardinality of Logs
Logs do not have time-series cardinality the way metrics do, but log backends still struggle with high-cardinality structured fields. Elasticsearch indexes balloon when a field has millions of unique values. Loki struggles with too many label combinations.
The common offenders:
request_idortrace_idas a Loki label. (Should be a body field, not a label.)user_idas a structured field in Elasticsearch with high concurrency. (Indexes inflate.)- Hostnames in environments with ephemeral instances. (Each new pod is a new label value.)
A quick audit of the top label values or top field cardinalities catches these. Move high-cardinality identifiers from labels to body, or drop them entirely if queries do not need them.
Compliance Carve-Outs
Audit and compliance logs have different rules. They typically cannot be sampled, dropped, or moved to cold tiers without proving accessibility for audit.
The cleanest answer is a separate pipeline for compliance-relevant streams. Route audit logs to a dedicated backend with full retention, full fields, and immutable storage. Treat operational logs with the cost discipline above. The two never mix.
This adds an integration step but keeps the audit conversation focused: "the operational log pipeline does not store the data audit asks about, and the audit log pipeline does, with the retention you require."
A Cost-Cutting Sequence
If you have an inflated log bill today and want to cut it without losing useful signal:
- Find the top 10 highest-volume log streams. Pick the noisiest one.
- Audit it. Identify lines that should not exist and remove them at source.
- Apply stratified sampling to the rest. Keep errors at 100%, sample the baseline at 5-20%.
- Drop fields nobody queries.
- Repeat on the next-noisiest stream.
- After the top streams are tamed, set up tiered storage and migrate old data to cold storage.
This sequence typically cuts log spend by 40-70% in a few weeks without changing what teams can actually answer from the data. The remaining savings come from steady discipline: every new service launches with a sampling policy, every code review catches new noisy log lines.
Log spend growing faster than headcount? We help teams design log pipelines that scale with traffic, not bills. scopeforged.com