Epismo
Get started

v2 Published snapshot

Validate a Solution Through a Proof of Concept

Use when an FDE or solution consultant must reduce a specific technical or business uncertainty through a bounded PoC and support a credible Go, Iterate, or No-Go decision. This covers test design through evidence review, not full production deployment. Do not confuse a polished demo or passing local script with proof of customer value.

Starting inputs

proposedSolution *

Candidate solution, integration or capability to test.

string

customer *

Customer or workstream identifier.

string

artifacts

Optional requirements, sample data, architecture, prior test reports.

array

constraints

Optional limitations: time, budget, security, data, environments, stakeholder access.

array

decisionQuestion *

Decision this PoC must make possible.

string

Steps

  1. 1

    Choose the highest-risk assumptions and decision rule

    Define the intended workflow and the few assumptions most able to invalidate the solution: user value, data adequacy, technical feasibility, quality, latency, reliability, security or economics. Compare them with existing customer evidence. Write explicit falsifiable hypotheses and thresholds for Go, Iterate and No-Go, including failure severity and the human decision maker. Explain why a cheaper non-build test would or would not suffice. Gate: test outcomes can meaningfully change a named decision before any build begins.

    V2 practical protocol:
    Precommit a decision matrix: hypothesis, success/failure threshold, severity of unacceptable failure, evidence sample, owner, evaluation date and action on result. For generative AI, distinguish task success, factual correctness, unauthorized disclosure, unsafe tool/action behavior, latency, cost per accepted result and required human review. Identify red-line errors (e.g. sensitive data leakage or irreversible unauthorized action) that invalidate Go regardless of mean scores. Document baseline manual/current-system performance and expected production conditions. Do not set thresholds retroactively after seeing results.

  2. 2

    Design a safe and representative evaluation

    Select representative scenarios, edge cases, realistic volume, negative tests, baseline alternative and ground-truth labeling. Specify metrics with units and denominators; for AI systems include hallucinations, unsafe actions and human-review requirements as appropriate. Document dataset provenance and access limits. Avoid production data movement or external service calls without authorized scope; synthetic data cannot establish real-world effectiveness by itself. Gate: the protocol would expose a meaningful failure rather than merely demonstrate a happy path.

    V2 practical protocol:
    Evaluation design: stratify cases by representative workflow, difficulty, user segment, language/format, error mode and edge conditions. Record sampling method, n per stratum, labeling rubric, ground-truth provenance, evaluator independence, adjudication method and acceptance rules for ambiguous cases. For factual answers, score supported/unsupported claims with source citations; for workflow automation, score end-to-end task completion and adverse side effects. Define denominators explicitly (e.g. harmful outputs / evaluated attempts) and record severity and confidence; tiny samples warrant uncertainty, not false precision. Include security, privacy, permissions and adversarial/negative cases relevant to risk. Synthetic examples may aid debugging but cannot by themselves validate production reliability.

  3. 3

    Build only enough to test the assumptions

    Implement or configure the smallest reversible working slice using available approved tools and access. Keep prototype shortcuts and missing production controls visible. Track deviations from evaluation protocol, dependencies and failure conditions. If execution is unavailable, provide a precise run sheet and expected observations instead of pretending to have run tests. Do not provision infrastructure, spend money or expose data outside existing authorization. Gate: the implementation is tied to predefined tests and limitations are explicit.

    V2 practical protocol:
    Execution ledger: baseline version, prototype code/config, evaluation dataset revision, model/system version, prompt/tool settings, dates, environment, authorization, test case IDs and actual observations. Prevent contamination by separating development examples from final held-out evaluation cases. Avoid test-set tuning without declaring a new held-out set. Capture costs and elapsed time where relevant, plus any failed runs, retries and manual interventions. If real execution is unavailable, mark each test "not run" and prepare a detailed run protocol; never represent a written test plan as a successful PoC.

  4. 4

    Evaluate results against the precommitted criteria

    Run authorized tests and preserve raw results, failure examples, timestamps, configuration and sample size. Compare each metric with its threshold, baseline and uncertainty; separate reproducible results from anecdotes. For AI, assess failure severity and where human intervention is necessary. Distinguish tests not run, tests failed and tests passed only in controlled conditions. Reject post-hoc threshold shifting without recording the change. Gate: material claims are independently inspectable and untested claims are not presented as validated.

    V2 practical protocol:
    Report evidence against each original threshold. For classification or factuality rates, show numerator, denominator, sampling frame, failure severity and confidence/uncertainty, not a naked percentage; compare with baseline under matching conditions. Include a failure taxonomy (data, retrieval, reasoning, permissions, user workflow, integration, environmental). Review false positives and false negatives when applicable, as well as cost/latency trade-offs. Separate "test failed", "test not run", "meets threshold with caveats" and "passes". Security or privacy red-line failures remain blockers regardless of average utility. Any changed threshold must be visibly labeled a protocol revision.

  5. 5

    Make the Go, Iterate, or No-Go recommendation

    Consolidate business value, technical findings, deployment gaps, expected cost and unresolved blockers. Compare options: progress to a time-boxed pilot, change scope and retest, or stop. Obtain the authorized sponsor's decision for any irreversible commitment; do not treat an agent recommendation as approval. Produce a concise decision report including reasons for the recommendation and the strongest counterargument. Gate: a sponsor can choose using evidence, limitations and explicit conditions rather than demo appeal.

    V2 practical protocol:
    Go/Iterate/No-Go rule: Go requires threshold evidence for all essential criteria, no unresolved red-line safety/privacy blockers, and credible production operating prerequisites; Iterate when a solvable failure has a bounded retest and meaningful potential; No-Go when value hypothesis fails, a structural constraint cannot be removed economically, or material criteria remain consistently unmet. A small sample may justify only a limited pilot, never broad production assurance. Decision brief must show per-hypothesis evidence, strongest failure case, test limitations, options, owner of final approval and exact additional proof required before production.