Splunk Agentic Ops · AgentOps Flight Recorder · Demo Artifact

Help ops teams investigate what AI agents actually did.

AgentOps Flight Recorder is for SRE, platform, security, and operations teams that need an audit trail when AI agents run tools. It turns agent actions into Splunk-ready evidence and uses evidence-bound AI to explain the incident without inventing facts.

index=agentops sourcetype="agentops:json"
| sort 0 _time
| table _time event.case_id event.event_id event.risk_level event.status event.summary

Who

SRE, platform, security, and operations teams that let AI agents touch tickets, tools, APIs, and release workflows.

Pain

After a bad action, a chat transcript is not enough. Teams need to search what the agent did, where risk appeared, and who approved the next step.

AI Use

The AI investigator is evidence-bound: it summarizes only from event IDs and SPL result rows, then asks a human to approve, reject, or hand off.

Value

Agent work becomes Splunk-ready operational evidence instead of an unreviewable conversation.

26Splunk-ready AgentOps events

3cases reconstructed from event trails

8medium/high/critical risk events

5human approval gates

$4.86estimated model/tool spend in sample

Run The Review Flow

This is the working prototype path: load event evidence, narrow to risk or approvals, run a Splunk-style search, then generate an evidence-bound AI investigation draft.

index=agentops sourcetype="agentops:json"
| sort 0 _time
| table _time event.case_id event.event_id event.risk_level event.status event.summary

Evidence-Bound AI Investigator

Choose a filter or run the AI summary. The assistant is constrained to cite event IDs from the evidence trail.

Case Overview

CASE-AI-OPS-001 critical

Generated a case handoff report with cited evidence IDs and final rejection decision.

11 events 3 approvals 1 blocked $0.0060

CASE-DFIR-002 high

Generated DFIR report with evidence IDs, unsupported claims, and human containment decision.

8 events 1 approvals 0 blocked $0.0120

CASE-CLOUD-003 medium

Closed cloud-agent workflow case with source citations, cost guardrail, and human approval captured.

7 events 1 approvals 0 blocked $4.8420

High-Risk And Blocked Actions

Event	Case	Risk	Status	Reason
evt-0005	CASE-AI-OPS-001	medium	failed	Regression test failed around duplicate retry suppression.
evt-0006	CASE-AI-OPS-001	high	warning	Production payment behavior change plus failing regression test.
evt-0007	CASE-AI-OPS-001	critical	blocked	Production deployment is destructive and requires human approval.
evt-0008	CASE-AI-OPS-001	medium	success	Human requested more evidence instead of approving the production deployment.
evt-0015	CASE-DFIR-002	medium	warning	Initial model answer overclaimed malware execution before process evidence was present.
evt-0016	CASE-DFIR-002	high	warning	Unsupported certainty detected in analyst draft.
evt-0017	CASE-DFIR-002	medium	success	Hash lookup returned low-confidence phishing kit association; raw indicator was redacted in report output.
evt-0024	CASE-CLOUD-003	medium	warning	Projected monthly model spend exceeded the manager's prototype budget.

Human Approval Queue

Event	Case	Decision	Summary
evt-0007	CASE-AI-OPS-001	blocked_by_policy	Attempted production deployment was blocked by policy before execution.
evt-0008	CASE-AI-OPS-001	needs_more_evidence	Human requested more evidence instead of approving the production deployment.
evt-0010	CASE-AI-OPS-001	rejected	Service owner rejected the release until the retry regression is fixed.
evt-0018	CASE-DFIR-002	approved	Human approved containment of the reported message and requested no endpoint isolation.
evt-0025	CASE-CLOUD-003	approved	Human approved the workflow with a rule that expensive model calls are escalation-only.

Saved SPL Searches

Timeline by case

index=agentops sourcetype="agentops:json"
| sort 0 _time
| table _time event.case_id event.event_id event.phase event.actor_type event.actor_name event.event_type event.status event.risk_level event.summary

High-risk or blocked actions

index=agentops sourcetype="agentops:json" (event.risk_level="high" OR event.risk_level="critical" OR event.status="blocked")
| table _time event.case_id event.event_id event.risk_level event.status event.risk_reason event.summary

Human approval queue

index=agentops sourcetype="agentops:json" event.human_approval_required=true
| table _time event.case_id event.event_id event.actor_name event.decision event.summary

Estimated model/tool cost by case

index=agentops sourcetype="agentops:json"
| stats sum(event.cost_usd_estimate) as estimated_cost by event.case_id
| sort - estimated_cost

Evidence Timeline

evt-0001 CASE-AI-OPS-001

task_start human / operations-lead

Human opened an AI-agent operations case after a release bot proposed a production change.

none

evt-0002 CASE-AI-OPS-001

plan_update ai_agent / codex-ops-agent

Agent created a read-only investigation plan with explicit approval gates before any production action.

low

evt-0003 CASE-AI-OPS-001

robot_task robot / uipath-evidence-robot

Robot gathered the change ticket, linked pull request, deployment window, and service owner.

low

evt-0004 CASE-AI-OPS-001

api_call api / github-api

API confirmed the pull request touches payment retry behavior and has one approving review.

low

evt-0005 CASE-AI-OPS-001

tool_call ai_agent / codex-ops-agent

Agent found a failing retry regression test that was not mentioned in the change ticket.

medium

evt-0006 CASE-AI-OPS-001

risk_signal ai_agent / codex-ops-agent

Case risk escalated because the proposed release affects payments and has failing verification.

high

evt-0007 CASE-AI-OPS-001

tool_call ai_agent / codex-ops-agent

Attempted production deployment was blocked by policy before execution.

critical

evt-0008 CASE-AI-OPS-001

approval_gate human / operations-lead

Human requested more evidence instead of approving the production deployment.

medium

evt-0009 CASE-AI-OPS-001

robot_task robot / uipath-evidence-robot

Robot requested service-owner signoff and attached the failed test evidence.

low

evt-0010 CASE-AI-OPS-001

approval_gate human / service-owner

Service owner rejected the release until the retry regression is fixed.

low

evt-0011 CASE-AI-OPS-001

handoff_report system / agentops-recorder

Generated a case handoff report with cited evidence IDs and final rejection decision.

none

evt-0012 CASE-DFIR-002

task_start human / security-analyst

Security analyst opened a suspicious email investigation with attached endpoint telemetry.

none

evt-0013 CASE-DFIR-002

plan_update ai_agent / dfir-analysis-agent

Agent proposed an evidence-first DFIR plan: preserve artifacts, list hypotheses, then verify each claim.

low

evt-0014 CASE-DFIR-002

evidence_captured robot / sift-collector

Collected email headers, attachment hash, browser download timeline, and endpoint process list.

low

evt-0015 CASE-DFIR-002

ai_call ai_agent / dfir-analysis-agent

Agent generated an initial hypothesis but flagged one claim as unsupported by current evidence.

medium

evt-0016 CASE-DFIR-002

risk_signal ai_agent / dfir-analysis-agent

System marked the draft finding as provisional because execution evidence was missing.

high

evt-0017 CASE-DFIR-002

api_call api / threat-intel-api

Hash lookup returned low-confidence phishing kit association; raw indicator was redacted in report output.

medium

evt-0018 CASE-DFIR-002

approval_gate human / security-analyst

Human approved containment of the reported message and requested no endpoint isolation.

low

evt-0019 CASE-DFIR-002

handoff_report system / agentops-recorder

Generated DFIR report with evidence IDs, unsupported claims, and human containment decision.

none

evt-0020 CASE-CLOUD-003

task_start human / support-manager

Support manager requested an agent that answers customer questions and escalates uncertain cases.

none

evt-0021 CASE-CLOUD-003

plan_update ai_agent / gemini-ops-navigator

Agent created a workflow plan using retrieval, MCP tools, cost guardrails, and human escalation.

low

evt-0022 CASE-CLOUD-003

api_call api / knowledge-base-mcp

MCP retrieval returned the current refund policy and source URL for citation.

low

evt-0023 CASE-CLOUD-003

ai_call ai_agent / gemini-ops-navigator

Agent drafted a customer answer grounded in the retrieved refund policy.

low

evt-0024 CASE-CLOUD-003

cost_signal ai_agent / gemini-ops-navigator

Cost guardrail warned that the high-quality model should be reserved for escalations.

medium

evt-0025 CASE-CLOUD-003

approval_gate human / support-manager

Human approved the workflow with a rule that expensive model calls are escalation-only.

low

evt-0026 CASE-CLOUD-003

task_end system / agentops-recorder

Closed cloud-agent workflow case with source citations, cost guardrail, and human approval captured.

none