AI Analyst Lab

Cohort 1

Bootcamp 2

Cohort 2

Bootcamp 1

36 free lessons on evaluating, measuring, and improving AI-powered products. Full slide decks with written explanations. Weekly quizzes and a certification exam. What you'll learn<ul><li>Design observable AI systems with proper logging and trace architecture</li><li>Run structured failure analysis on production data</li><li>Translate AI quality metrics to user behavior and business results</li><li>Build evaluation workflows from offline tests through production monitoring</li><li>Make ship/iterate/pause/rollback decisions based on evaluation data</li></ul>

Free 6-week AI Evals Certification Course

Leave with a working AI analyst system on your laptop (skills, agents, data connections) and a finished analysis ready for Monday.

Principal Data Scientist @ Ontra 

Shane Butler

Principal Data Scientist @ Ontra | Ex-Stripe, Nextdoor, PwC, Appfolio

Data Science Leader |Ex-Microsoft

Sravya Madipalli

Data Science Leader@Superhuman (Prev. Grammarly)| Ex-Microsoft, eBay, Nextdoor

Head of Data @ Ontra AI | Ex-LinkedIn

Hai Guan

Head of Data @ Ontra | Ex-Nextdoor, LinkedIn, Pinterest, Meta

Build AI Analysts in Claude Code

Sr. Manager, Collections Analytics

Data Scientist

Lead Data Analyst

Vice President

Director, Business Operations & Analytics

product data scientist

Senior Data Scientisit

Sr. Analytics Manager

Engineering Manager

This is a hands-on weekend bootcamp (4 hours Saturday, 4 hours Sunday) where you build AI analysts powered by Claude Code. Not a lecture series. Not a demo you watch. You set up your own environment, build custom skills and agents, connect to real data, and leave with a working system that turns business questions into charts, narratives, and presentations in minutes.No Python. No SQL. No coding of any kind. Claude Code writes all the code. You direct the analysis.By Sunday evening, you will have a complete AI analyst repo on your computer: 45 skills, 27 agents, data connections, and a finished analysis you can share with your manager Monday morning. Everything runs locally. No vendor lock-in. No proprietary platform. Just a repo you own.AI is changing how teams make decisions with data. The bottleneck is no longer the analysis — it is knowing how to set up the tools. This bootcamp gives you that setup in a weekend. You will not learn about agentic systems. You will build one.

Build agentic data analysts for your company in one weekend. No code.

Set up Claude Code as your personal AI analyst environment

Build custom skills that encode analytics best practices as markdown

 Build agents that automate multi-step analysis workflows

Produce a complete analysis with charts, narrative, and slide deck

Connect your own company data and analyze it before you leave

Product managers who are tired of waiting in the data team's queue. Stop waiting. Analyze it yourself.

Data scientists who want to multiply their output. You know how to do the analysis manually. Now automate it.

Engineers who want to make data-driven decisions without hiring a data team.

Transform from consumer of analytics to independent operator - asking sharp questions, validating answers, shipping decisions.

Stripe

Nextdoor

PwC India

Ontra

AppFolio

Microsoft

eBay

Grammarly

Superhuman

LinkedIn

Pinterest

Ask Better Questions, Make Faster Decisions - with AI as Your Analyst

Frame Problems Like a Senior Data Scientist

Design Metrics That Actually Drive Decisions

Diagnose Root Causes Without Guessing

Design Experiments That Ship

Turn Analysis Into Action With AI as Your Copilot

Synthesize Insights and Prioritize for Impact

Product Managers, engineers, and operators who need analytical independence and frameworks to measure success and drive decisions.

Designers and researchers stepping into strategy roles who need to quantify impact, run experiments, and influence with data-driven insights

Data professionals who want to frame better questions, delegate technical execution to AI, and drive impact with agentic analytics tools.

Learn practical, hands-on methods to evaluate, measure, and improve AI products using data science to make better product decisions.

Principal Data Scientist, AI Evaluations

Principal Data Scientist, AI Evaluations at Ontra

With the recent leaps in model capabilities, it is becoming clear that much of the AI evaluation work previously requiring humans to build and run can be automated through agentic systems. This course teaches you how to build those systems, and where human judgment and input is still required.The foundational work of designing eval frameworks, defining quality signals, and structuring failure analysis remains critical. But the execution of that work, running evaluations, scoring results, iterating on improvements, and surfacing what matters, can now be handled by agentic workflows that operate continuously and at scale.This course equips you with the capability to build, test, and deploy automated evaluation systems using Claude Code. You will leave with working systems, not just conceptual understanding.The focus is on automation that is trustworthy. Systems that score their own output, flag uncertainty, and know when to surface decisions to a human rather than proceeding autonomously.The foundational content on AI evals for product development is available free at <a target="_blank" rel="noopener noreferrer nofollow" href="https://aianalystlab.ai">aianalystlab.ai</a>. If you are new to AI evals, start there.

Build the agentic systems that automate your AI evaluation workflow from end to 

Build agentic evaluation workflows in Claude Code

Automate scoring and quality assessment

Implement iterative improvement loops

Scale experiment execution

Integrate with production data and decision-making

You will get the most out of it if you are actively working on, or about to ship, AI features in a real product.

Real-World AI Product Experience

Product Managers: Want automated eval systems for your team? Learn what is possible, reliable, and where human oversight stays essential.<pre><code class="language-markdown"></code></pre>

Data Scientists &amp; Analysts: Spending more time running evals than learning? Build systems that handle execution so you focus on decisions.

Engineers and ML Engineers: val bottlenecks slowing your release cycle? Automate repetitive work and ship with speed and confidence.

Vibe Code Annotation UIs for AI Analytics Evals

Know when you need annotations for ground truth, production review, and continuous discovery.

Identify where annotation fits in the AI evaluation lifecyle

Choose the right interface patterns so reviewers can judge quality without fighting the UI.

Design annotation tasks that match the evaluator’s workload

Specify what the UI must show, what data it reads and writes, and how to standardize to make work is repeatable.

Define a lightweight “custom UI” blueprint that scales

Learn Agentic Analytics workflows and AI Eval design. Get feedback on what you’re building from other AI builders.

Join AI Builders Slack Community. Link: bit.ly/ai-connect

Root Cause Analysis in Claude Code

Senior DS Leader (Ex-Microsoft)

Head of Data at Ontra, Ex-LinkedIn

Use Level 0-to-N breakdown to go from 'revenue dropped' to the specific segment and cause driving the change.

Decompose metric drops into root causes

Watch Claude Code peel back layers of a metric drop to find the exact driver. No SQL, no pivot tables.

Run a full decomposition in Claude Code

Leave with a reusable decomposition prompt that works on any metric drop you need to investigate.

Build a root cause template for your metrics

Validate Claude Code Analytics Output

Principal Data Scientist at Ontra

Check structure, logic, business rules, and segment-level reversals before trusting the output.

Run a 4-layer validation on any AI analysis

For each step, specify what evidence would justify ship, iterate, or stop, and what “evidence” is misleading.

Define evidence that supports a ship decision

Learn the specific failure modes where AI confidently produces wrong answers and how to spot them.

Catch errors AI analysis gets wrong

Use Claude Code to approach the same question a different way to measure both capability and reliability

Re-derive findings to confirm correctness

Turn Insights to Action in Claude Code

Use the Close-the-Loop checklist to turn every recommendation into an owned, measurable next step.

Build action plans from analysis findings

Watch Claude Code take analysis findings and produce action plans with owners, metrics, and deadlines.

Generate structured plans with Claude Code

Add baselines, targets, check-back dates, and fallback plans so every recommendation has teeth.

Stop insights from dying after the meeting

Build Slide Decks in Claude Code

Watch Claude Code take a dataset and produce slides with charts, speaker notes, and recommendations.

Go from raw data to a themed slide deck

Learn how Marp themes work and how to iterate on deck design without writing CSS or touching code.

Apply and customize presentation themes

Follow the 6-step pipeline: data, story structure, slide content, charts, theme, and final deck.

Use the deck-building pipeline end to end

Analyze Metric Trade-offs in Claude Code

Learn which guardrail to attach to common success metrics so you catch trade-offs before they ship.

Pair success metrics with the right guardrails

Know when a win on your success metric is hiding a loss. Conversion up but AOV down. Activation up but churn spiking.

Detect trade-offs in experiment results

Watch Claude Code analyze an experiment, flag guardrail violations, and surface the trade-offs automatically.

Learn directly from AI Analyst Lab

AI Analyst Lab

Courses

Build AI Analysts in Claude Code

AI Analytics for Builders

Automate AI Evals with Claude Code

Lightning Lessons