Everybody is talking about AI. But nobody shows you how to use it for Finance. Pioneer Finance Professionals using ChatGPT, Copilot, Gemini and Python are already boosting their career. How?They are more productive.Their emails and writing look more professionals.They showcase more valuable input to management. You want to stay relevant and secure your career? Join this cohort to be part of the innovative group of Finance Professionals who want to leverage AI for their career. This is the only course to give you the methodology and the use cases to use ChatGPT / Copilot / Gemini for Finance and Accounting.&nbsp;After this course you will be able to do the following: • Give advanced instructions to Generative AI• Use Generative AI for Financial Analysis• Solve the Confidential Data issue• Build your own GPT for Finance• Use Python with Generative AI&nbsp;In this course we show you our secret method to make Financial Analysis on your own data while still using the power of Generative AI.&nbsp;&nbsp;New:We will cover GPT Builder, the underrated function to build your own GPT tailored with your data and documents.Learn how to build your own AI GPT in less than 15 minutes.&nbsp;We will cover the "Advanced Data Analysis" from ChatGPT 4o and ChatGPT 4:With this function you can upload, analyze, edit and download Excel files. We will also provide a Python Learning Training Path for absolute beginners complemented with a collection of 31 Python notebook exercises, useful for finance professionals. Dates:Tuesday 24 September 2024Thursday 26 September 2024 When:15:00 PM New York Time (EDT)21:00 PM Berlin Time (CEST)12:00 PM Los Angeles Time (PT)

Leverage AI for Finance Analysis to Boost Productivity & Accelerate your Career

<ul><li>Recap of ChatGPT’s input and output formats</li><li>Fine-tuning ChatGPT for domain-specific applications in FP&amp;A</li><li>Advanced text generation techniques using ChatGPT, including controlling output, conditional generation, and prompt engineering</li></ul>

ChatGPT Fundamentals

<ul><li>Overview of Python and ChatGPT for FP&amp;A professionals</li><li>Introduction to Google Colab, a cloud-based Jupyter notebook environment for Python</li><li>Setting up Python environment in Google Colab and familiarizing with its features</li></ul>

Introduction to Python, ChatGPT, and Google Colab

<ul><li>Basics of Python syntax, data types, variables, and basic operations</li><li>Python Libraries (what are they and how to use them)</li><li>Advanced data manipulation with Pandas, including data filtering, aggregation, and transformation</li><li>Exploring advanced NumPy features for numerical computing in FP&amp;A</li></ul>

Python Basics for FP&A Professionals

<ul><li>Quick Introduction to PVM analysis as a key technique in financial performance analysis</li><li>Hands-on exercises for calculating and interpreting PVM effects using Python and Pandas</li><li>Advanced PVM analysis techniques, such as sensitivity analysis and visualization using Python libraries</li></ul>

PVM (Price-Volume-Mix) Analysis with Python and ChatGPT

<ul><li>Overview of three-way financial models for integrated financial statements (Income Statement, Balance Sheet, and Cash Flow Statement)</li><li>Hands-on exercises for building three-way models using Python and Pandas</li></ul>

Building a Three-Way Model with Python & ChatGPT

<ul><li>Recap of the key topics covered in the course</li><li>Tips for further exploration and advanced applications of Python and ChatGPT in FP&amp;A</li><li>Final Q&amp;A session to address any remaining questions or concerns</li></ul>

Conclusion and Next Steps

We will teach you how to upload Excel files into ChatGPT, edit, analyze them and download new Excel files.

New: Analyze Excel files in ChatGPT


We will teach you how to use ChatGPT Pluggins and get the best out of it.

New: ChatGPT Plugins


-ChatGPT for Finance Guide (PDF Guide of 92 pages with 22 use cases for Finance)-1 hour recorded video tutorial to explain you how to start ChatGPT

BONUS: ChatGPT for beginners

Course syllabus

New York Time: 15:00 ETBerlin Time: 21:00 CETLos Angeles Time: 12:00PM PT﻿<ul><li>ChatGPT Fundamentals</li><li>ChatGPT 4</li><li>Create your GPT</li><li>Introduction to Python, ChatGPT, and Google Colab</li><li>Python Basics for FP&amp;A Professionals</li></ul>

Tuesday 24 September, 2024 (2 hours)

New York Time: 15:00 ETBerlin Time: 21:00 CETLos Angeles Time: 12:00PM PT <ul><li>Financial Analysis with Python &amp; ChatGPT</li><li>Automate Finance with Python &amp; ChatGPT</li><li>Conclusion and Next Steps</li></ul>

Thursday 26 September, 2024 (2 hours)

<ul><li>How to start using ChatGPT</li><li>Real use cases for Finance</li></ul> If you have never used ChatGPT, you can go through this video course to get ready for the advanced part!

ChatGPT for Finance Video Course + PDF

Course schedule

You want to use ChatGPT / Copilot / Gemini for Financial Analysis

You have already used ChatGPT / Copilot / Gemini but want to go to the next level

You want to be a pioneer in using technology and Generative AI to boost your work in Finance

Who is this course for

What you’ll get out of this course

We will train you to write complex instructions to ChatGPT / Copilot / Gemini. Once you know how to write to ChatGPT, you will be able to use as your assistant and second brain for finance tasks.

Know how to give advanced instructions to ChatGPT / Copilot / Gemini

You want to know what is the impact on your margin when you change sales volume or the price of product? Learn how to use ChatGPT to perform quick analysis.

Speed-up your financial analysis

You will learn how to use Python for financial analysis without having to learn how to code! This is becoming important for all Finance professionals because this is the language ChatGPT is using when doing its Financial analysis!

Use Python for Financial Analysis without Learning How to Code

Learn how to connect ChatGPT to the Internet and how to use the ChatGPT GPTs.

Connect ChatGPT to the internet

You will get our secret method to make financial and data analysis on your confidential data while still using the power of ChatGPT / Copilot / Gemini.

Learn our secret method to work on confidential data

What you will get: • 4h Live Course via Zoom with 2 experts ($899 value)• 2.5h Self Paced Video Course ($197 value)• PDF ChatGPT for Finance ($97 value)• Python Crash Course ($199 value)• Companion Booklet ($47 value)• 450 prompts ($67 value) Total Value: $1506Price: $599

Save $907 if you enroll now

Learn Advanced Financial Analysis with Generative AI (ChatGPT, Copilot, Gemini and Python)

Nicolas Boucher

Finance Thought Leader with 1 million followers on social media and 15+ years' experience in multinational groups & leading auditing firms

Christian Martinez

Finance Automation Manager at Kraft Heinz Founder of The Financial Fox

Cohort 8

Cohort 9

The Mahesh Yadav Institute focuses on quality over quantity and is passionate to educate the next generation of AI product leaders. We only offer 4 cohorts throughout the year and limit each cohort to 40-50 students to maximize the 1:1 mentoring time you get from Mahesh. 📚&nbsp;We are part of Maven's BUILDWITH AI series, where only the Top instructors in AI are selected. In terms of leadership style, Mahesh is a servant leader. He believes in giving back and wants to help you to become #1 in the AI/ML space. For the general public, we have a Free Sat community, where they can come and learn AI, meet industry speakers and ask Mahesh questions LIVE. You can join our FREE AI PM group here: https://www.linkedin.com/company/mahesh-ai-pm-community 🚀 Why Join Us?For Aspiring AI Leaders:- Access 5 live sessions (90 mins each) covering advanced AI/ML strategies.- Gain hands-on experience through 5 lab sessions (60-90 mins) – crucial if you want to stand out as an AI Product Manager.- Early access to essential course materials to jumpstart your learning journey.- Lifetime support: Whether you're building your own AI company, seeking mentorship, or transitioning to an AI PM role, Mahesh will continue to guide you post-graduation. 🏆 Our Success Stats :- All alumni reports an increase in confidence to lead AI strategy, build an AI company or ship an AI-powered product.- 5 alumni from recent cohorts got AI PM jobs or have started their own companies.- Cohort 1 members have started getting AI PM interviews. We believe this is only achievable in a smaller class setting and genuine care for students in their career journey post-graduation. 👨‍💻 𝗔𝗕𝗢𝗨𝗧 𝗧𝗛𝗘 𝗜𝗡𝗦𝗧𝗥𝗨𝗖𝗧𝗢𝗥 Mahesh Yadav is the GenAI PM lead at Google GenAI team. He has over 20 years of experience in AI/ML Product management and engineering, specializing in deploying large language model (LLM) based services and enhancing AI infrastructure at companies like&nbsp;AWS AI (Bedrock), Meta (PyTorch and Feed), and Microsoft Azure AI.&nbsp;Mahesh is&nbsp;the original product builder for AWS Bedrock GenAI services, and he grew the platform's customers from zero to 10k in 8 months. In addition, he optimized Pytorch at Meta by moving it from a single cluster training platform to a multi-cluster training platform and expanded the feed models from Billion to Trillion in scale to increase feed CTR. Mahesh is on a mission to educate the next generation of AI/Ml product managers. He has taught 1000s of students in his Meta internal channel course, and 5 cohorts at PMExercises. He has helped hundreds of PMs to interview for AI/ML roles for the past 10 years.Mahesh's vision is to educate the next wave of AI product managers by sharing his successes and mistakes—ensuring you're prepared for the challenges ahead. 💡 𝗔𝗕𝗢𝗨𝗧 𝗧𝗛𝗘 𝗖𝗢𝗨𝗥𝗦𝗘 Drawing from my experiences developing ML/AI products and working at leading companies like Facebook, AWS, and Google, I've observed a critical gap. Many aspiring AI PMs struggle in the AI space due to a lack of understanding of AI/PM product basics, technical concepts, or the absence of a hands-on project or portfolio that demonstrates their AI PM skills in the market. In this course, I will share in-depth topics that will truly help set you apart for AI ML interviews, and rehash what you already know. In addition, we will provide a hands-on lab to build your AI product portfolio. I have designed this course based on my two decades of experience at Microsoft, Meta, Amazon &amp; Google to not only provide deep insights into AI PM fundamentals but also to focus on the technical aspects and practical applications needed in the AI field which will&nbsp;prepare you for the real-life interview settings, and your AI PM career. Throughout the course, you will have practical lab sessions to help you to understand the tradeoff you have to make as an AI product manager. Furthermore, you will also develop effective product requirement documents (PRDs), and embark on creating your first AI-driven product with a talented community of highly driven individuals, guided by me. Key AI PM Technical Knowledge Covered (This is a small sample of what we will go through in class. AI is a rapidly changing field. What set me apart from other instructors is I am in the core GenAI team working with the core tech, so I can continue to bring in the latest concepts and industry insights for each cohort)• Basic AI Concepts (e.g. Neural Networks, Transformers, Diffusion)• Real-world examples of ML/AL execution and tradeoffs• Special focus on GenAI, its applications, and deployment with technical deep-dive• Types of AI Models and Applications• How to evaluate the success of GenAI models and systems• AI/ML Stacks• Advance knowledge and session on AGI's• LLM-Powered Agents and Agent Creation• Prompt Engineering• Prompting Techniques (e.g. Zero-shot, chain of thoughts)• Fine-tuning ML models (e.g. PEFT, LoRA) &amp; RLHF• Retrieval Augmented Generation (RAG) &amp; Retrieval Augmented Fine-Tuning (RAFT) 💡&nbsp;COURSE PREREQUISITESThis course requires a traditional PM or equivalent background, but we have cohort members from other backgrounds in design and ML engineering who have proved to be a great fit too. Contact us if you have questions if this course will be a good fit for you.

Master the AI PM technical essentials and get real-world experience in GenAI

Provide a brief description for each module as a way to sell the value of what students will be learning throughout the course.

First Module

Second Module

Third Module

Recordings will be provided in the student portal if you cannot join live.

5 Saturdays (Aug 24th, Aug 31th, Sept 7th, Sep 14th, Sep 21st)

You will have 4 lab sessions to build with Mahesh. It is also our weekly open office hour.

Wed Lab Sessions/Optional Q&A

We want to help you to continue your learning journey even after the cohort ends. We meet weekly on my LinkedIn community. Current and previous cohort members join us to keep learning. Join us for a free event:https://www.linkedin.com/company/mahesh-ai-pm-community

Weekly Community Sessions

Traditional PM looking to transition into AI Product Management. This course will help you demystify GenAI.

Current AI PM’s looking to upskill and build hands-on projects and portfolios to set themselves apart in the current market.

AI/ML data scientist / cloud architects/UX and program managers transitioning into product management.

Entrepreneurs and product builders looking to build and improve their GenAI products.

Master the essentials and patterns unique to AI(GenAI) product development. You can then position yourself for that new AI PM job based on your past experience and newly acquired skills.

Learn Core AI Concepts and Position yourself as an AI Product Manager

Acquired knowledge needed to identify new opportunities in AI/ML, develop PRDs, and decide when to use an AI based solution.

Solving AI Product Sense Questions

Throughout the course, Mahesh will walk you through the most common types of GenAI systems and AI models. Expect a deep-dive in Large Language Models (LLMs). We will also go over common AI systems such as Computer Vision, Regression, Search and Recommendation etc.

Pass that AI PM technical interview

Learn from Experience: Get insights from real AI product case studies from Mahesh's journey, including bumps along the road.

How to Handle AI Product Management Case Studies.

How to convince stakeholders to invest in your AI features and products? What mistakes to avoid? How to develop MVP in GenAI, and how to come up with pricing strategies?

Advancing your AI Product Management Skills

Create your own GenAI product or collaborate with a startup, enhancing your portfolio.

Hands-On Building: Build a portfolio of GenAI powered products.

Mahesh will coach you in a small group settings to guide your PRD and (or) Capstone project to succeed.

Coaching sessions with Mahesh on your PRD/capstone project

#1 AI product management course for educating Silicon Valley's best PMs to succeed in GenAI product leadership

Cohort 3

Mahesh Yadav

Cohort 2

Cohort 4

**Welcome to Cohort 3! 109 People Enrolled For Cohort 1&amp;2 ﻿What You’ll Achieve in Just 4 Weeks: Get ready to go from basic to start building agentic workflows! By the end of this course, you’ll confidently implement AI to supercharge your daily operations which will give you the fundamental skills to understanding how to scale a business with AI – all with your own custom AI tutor, “The Professor,” as your guide 🤖 (adapted to your best learning &amp; communication style!) What You’ll Learn: - Essential AI fundamentals for business growth, including AI's role in different business areas and ethical considerations&nbsp;- Key AI concepts, platforms, and tools, along with their strengths and limitations&nbsp;- How to create and use a Business Second Brain and customize your AI tutor, "The Professor"&nbsp;to suit your learning &amp; communication style- Fundamentals of prompt engineering and advanced techniques for multi-step workflows&nbsp;- Strategies to align AI-generated content with your brand voice&nbsp;- Advanced prompt engineering for AI assistants and integration with your knowledge base&nbsp;- Refining your outputs with Evals- Basics of AI workflow design for business efficiency What You’ll Be Able to Do: - Identify AI opportunities within your business and align them with growth strategies&nbsp;- Craft effective prompts and build a prompt library tailored to your needs&nbsp;- Create a brand voice snippet to ensure consistency in AI-generated content&nbsp;- Build and prepare an AI assistant for deployment in your business&nbsp;- Integrate AI assistants with your knowledge base and Business Second Brain&nbsp;- Preview basic AI workflows &amp; how they can be implemented into businesses What You’ll Have Access To: • Hands-on projects with leading AI tools like Cassidy.ai - we cover the cost!•The chance to reconstruct “The Professor” and learn how to repurpose it into any role you need – from customer service to internal knowledge management.•	A personalized prompt library tailored to your specific needs.•	The Notion AI for Business Growth Guidebook – your go-to resource.•	Actionable templates, checklists, and resources for seamless AI integration.•	A prompt library populated with some of our prompts to get you started! This course is your launchpad. After 4 weeks, you’ll be primed and ready for our program, “How to Scale a Business with AI Workflows - Accelerator” There, you’ll learn more advanced skills to deploy AI workflows that drive real ROI at scale - and graduate with your own AI Workflows ready to deploy for your own work or for clients.

From Basic Beginner to Confident AI Practitioner -  Zero Coding Required

This will be our live learning &amp; Q&amp;A time - join one, or both or catch up with the recording.

Tuesdays 

Fridays

Learning with us is all about practical application. We want you to to be able to apply what you learn back into your work &amp; business!

Weekly projects

Non tech-professionals wanting to become the AI leaders in the workplace

Consultants and agencies looking to offer AI consulting and services

Entrepreneurs &amp; business owners looking to build businesses as scalable, growth machines

You will immerse yourself in understanding how AI can revolutionize various business areas and appreciate its strategic implications for growth and competitive advantage

Deep Dive Into AI for Business Growth:

Build a solid foundation in AI essentials and explore a range of no-code AI tools (Cassidy AI, Respell) to kickstart your journey toward AI integration without needing a technical background.

Gain a Robust Foundation in AI Concepts and No-Code Tools

Develop the skills to craft and refine effective prompts for AI platforms, building a versatile prompt library that caters to your specific business needs and scenarios

Hands-On Prompt Engineering and Customizing Your Prompt Library

Learn techniques to train AI to reflect your brand's tone consistently across various communication channels, ensuring AI outputs feel personal and authentic to your audience

Know How To Get AI To Sound Like You With Your Brand Voice Snippet

Engage with an interactive AI tutor that adapts to your learning and communication style. 'The Professor' will assist you throughout the course, offering tailored support and enhancing your understanding of AI concepts, prompt crafting, and working on your projects.

Co-pilot Your Learning With A Custom AI Tutor - "The Professor"

Gain practical experience in configuring and deploying specialized AI assistants designed to enhance efficiency and productivity within your daily operations

Deploy the first versions of your AI assistants

Learn how to refine and optimize your outputs with evals that lead to scalable workflows

Gain a 1% AI Skill

Get a preview of how to construct agentic workflows for business efficiency that lay the groundwork for future scaling

Gain an introduction to scaling with agentic workflows

*Your own AI tutor* + foundations of our frameworks, playbooks, templates to build AI workflows that add 6-8+ fig to a business bottom line

Sara Davison

Founder & CEO of Maivenly

Tyler Fisk

Founder & Chief Nerd of Galactic Ranch

Cohort 1

Learn how to use machine learning techniques for predicting future outcomes in time series to optimize business processes. The course features practical lessons heavily that we derived from two decades of working on some of the world's hardest forecasting problems at Amazon retail, Zalando and for AWS and its customers. You'll pick up the necessary theory, get hands-on example and learn about the tricks of the forecasting trade.

What you will learn

<ul><li>Understand which and how business processes can be optimized by incorporating (probabilistic) predictions of future outcomes</li><li>Differentiate strategic from operational forecasting problems with examples from Zalando and Amazon</li><li>Measure and compare the accuracy of different forecasts</li></ul>

Session 1: Should Your Business Problem Be Solved With Forecasting?

<ul><li>Case Study: Forecasting top-level energy demand and prices</li><li>Understand the underlying business problem and the challenges of the resulting data-constrained forecasting problem</li><li>Identify the effects and structural components that make up the data, such as trend(s), seasonality, exogenous shocks, and noise</li><li>Identify the appropriate method and tool, such</li></ul>

Session 2: Forecasting Solutions Using a Small Set of Time Series

 

<ul><li>Case Study: Retail demand forecasting</li><li>Build an intuition for the data via visualization of individual time series and aggregate summaries</li><li>Obtain co-variates/features and process them</li><li>Use and tune global ML-powered methods such as Gradient Boosted Trees and Neural Network-based methods like DeepAR</li></ul>

Session 3: Forecasting Solutions With a Large Set of Time Series

 

<ul><li>Case Study: Forecasting with causal inputs</li><li>Forecast demand subject to price changes for millions of products</li><li>Build what-if analysis using simple and advanced approaches</li><li>Evaluate &amp; improve forecasting in counterfactual situations</li></ul>

Session 4: Forecasting Solutions With Dependency Structures

 

<ul><li>Practical tactics for forecasting exemplified by labor planning</li><li>Productionize forecasting models including retraining schemes</li><li>Handle missing data and the associated perils</li><li>Research approaches to outliers/extreme events such as blizzards and pandemics</li></ul>

Session 5: What Best Practices Help You Avoid Common Pitfalls in Production?

We have 5 full sessions each 2h, starting on Monday 4 March, 12pm EST (6pm CET). Sessions will consist of an engaging mix of presentations, activities and notebooks.

Session Timing & Details

We'll have office hours, a deep dive with Max Mergenthaler, Nixtla and guest speaker Sercan Arik (Google). Past guest speakers:<ul><li>Slawek Smyl (M4 winner; Walmart, Meta, Uber, Microsoft),</li><li>Sean Taylor (Prophet; Lyft, Meta)</li><li>Boris Oreshkin (NBEATS, Amazon)</li></ul>

Additional activities

Course schedule for March 2024

Data Scientists who want to go beyond standard ML/AI problems and solve forecasting related business problems

Business analysts with familiarity in machine learning in industry settings who want to uplevel themselves in a top ML application domain

Economists and Applied Scientists who want to apply industry-proven modern forecasting techniques

<ul><li>Understand which and how business processes can be optimized by (probabilistic) forecasts</li><li>Differentiate strategic from operational forecasting problems with real-world examples from Zalando and Amazon</li><li>Measure and compare the accuracy of different forecasts</li></ul>

Identify the business problems that can benefit from modern time series forecasting techniques
 

<ul><li>obtain the right co-variates for your problems and learn how to process them</li><li>pick the appropriate model family (and software implementation) for your forecasting challenge</li><li>enable model iterations through tools such as state of the art hyper parameter optimization</li></ul>

Apply appropriate forecasting techniques to maximize the effectiveness of your solution
 

<ul><li>Visualize results in meaningful ways to make your efforts tangible to a non-technical audience</li><li>Understand the right quantitative evaluations to ensure that your results are convincing</li></ul>

Develop the skills to present results effectively to persuade a non-technical audience 
 

Get hands-on experience with modern forecasting tools & learn from case studies of the toughest forecasting challenges in the industry

Tim Januschowski

Director of Engineering, Databricks

Jan Gasthaus

Software Engineer, Meta

Cohort Q3 2024

Cohort Q1 2025

Become the RAG Product Thinker Your Company Needs In the rapidly evolving world of AI, companies are in a fierce arms race. They're scrambling to find developers and product leaders who can successfully incorporate AI into their products before competitors do it better. Are you ready to be that leader? The Challenge: Navigating the RAG Black Box You're working on AI-powered applications, but: * Time and resources are limited* The best path forward isn't clear* The inner workings of your product are shrouded in uncertainty The Solution: Develop the "Product Sense" for AI In just 6 weeks, learn to: * Optimize search quality and latency* Design robust feedback loops for continuous improvement* Implement data-driven strategies for maximum impact* Navigate AI product decisions in uncertain environments Course Breakdown: From Fundamentals to Advanced Techniques Week 1: Fundamentals and Synthetic Data Generation* Understand the importance of having a system for evaluating and improving RAG* Learn to generate synthetic data for fast evaluations* Master precision and recall metrics for retrieval evaluation* Distinguish between leading and lagging metrics* Implement the RAG System Inference Flywheel Week 2: Segmentation and AnalysisLearn the importance of segmenting queries and usersDifferentiate between inventory issues and capability issuesUse clustering and classification to identify query types and segmentsSet up dashboards to monitor query distributions and performance over timePrioritize improvements based on impact, volume, and likelihood of success Week 3: Structured Extraction and MultimodalityLearn techniques for handling different data types: documents, images, tablesExtract metadata and generate synthetic data/summaries to improve searchImplement approaches for document search, image search, and querying tablesApply Week 1-2 techniques to evaluate and improve individual indicesBalance trade-offs between generalized and specialized approaches in multimodal RAG Week 4: Query Routing and Tool SelectionCombine multiple search indices into a cohesive applicationImplement parallel function calling and API gateways to route queriesEvaluate tool selection as a classification taskAvoid data leakage when generating examplesBreak down probability of success into tool selection and retrieval components Week 5: Representations and Fine-tuningUnderstand limitations of pre-trained embedding modelsLearn the importance of fine-tuning embeddings and re-rankers on domain-specific dataCollect relevance data to create triplet training examplesExplore benefits of fine-tuning a single model across multiple tasksDiscover how relatively small amounts of data can lead to significant improvements Week 6: Product Design and User ExperienceImplement techniques for collecting user feedbackDesign streaming strategies to improve perceived latencyCreate UI components for rendering citations and follow-up actionsApply prompting techniques like chain-of-thought and monologuesAdd validators to improve reliability and response qualityDevelop strategies for handling negative examples Why This Course, Why Now?RAG is becoming essential for competitive AI integrationFocus is shifting from basic implementation to performance optimizationCore principles of effective RAG systems are crystallizingEarly adopters gain significant market advantages What You'll LearnCold start your evaluation pipeline for retrievalUnderstand the limitations of embedding models and how to think about rerankers and fine-tuningMaster retrieval metrics and use them to quickly run experimentsIdentify high-impact tasks and prioritize effectivelyMake informed tradeoffs and choose relevant metricsFocus on what matters most in AI product development Why Your Team Needs ThisAlign on RAG best practicesSave months of trial and errorBuild scalable systems that prevent future rewritesFoster a data-driven culture of continuous improvementBridge gaps between technical and business teamsGain competitive edge in AI implementationJustify AI investments to leadershipLearn from real-world case studies across industries About Your InstructorsJason Liu is a seasoned machine learning consultant with experience at Stitchfix, and Meta, Stitch Fix, and dozens more companies, ranging from startups to Limitless to Larger enterprises like Hubspot and Zzapier. When companies struggle to make progress, they hire Jason help their AI teams find "the path" forward. Jason Liu is a machine learning engineer and data scientist with 8 years of experience in building recommendation systems and multi-modal semantic search products at Stitchfix What You'll GetA community of other operators and AI product thinkersHands-on experience with real-world RAG applicationsStrategies for continuous system improvementSkills applicable to all AI initiatives, not just RAGAccess to instructors' real-world experience and immediately applicable insightsBonus Offers (Over $1500 in Value!)$500 Cohere credits$200 LanceDB credits and free access to Lance Cloud$500 in Modal Labs credits6 months free Notion AI Plus3 months Braintrust access ($250 value) Who Should Apply?This course is perfect for you if:You're working on software products incorporating AIYou've built a RAG prototype and want to move to productionYou're looking to develop a "product sense" for AI applications Won't a lot change by February?Course focuses on enduring principles, not just current toolsYou'll learn to evaluate and integrate new technologies rapidlyStrategies taught focus on ongoing system optimizationBy February, you'll be positioned to leverage new developments immediatelyCourse content will be updated to reflect any significant changesSkills developed are foundational and will remain relevant Don't Miss Out!🚀 Limited Spots Available 🚀 Our small-group cohorts fill up fast. Don't wait to level up your RAG skills. Remember, you only pay after your application is approved. We're so confident in the value of this course that we offer a full refund if you don't see meaningful improvements in your processes within 5 weeks. Join a small cohort of other teams shipping real applications and take your RAG skills to the next level. Don't do it alone - be part of a community of professionals facing similar challenges and working towards AI excellence.

Acquire the skills & confidently improving and iterate on RAG applications

Workshops covering each step of the playbook and helping you build process improvements in your RAG application

Tuesday: Workshops

The first half hour will be interactive breakout sessions, and the closing half-hour each week is Q&amp;A

Wednesday: Office Hours + Breakout Sessions

Guest instructors covering key topics in both innovative theory and practical applications in RAG system development.

Thursday: Guest Speakers

Course Schedule Each Week

An Engineering or Product leader looking to improve an existing RAG system MVP

Solving problems like poor retrieval, unreliable outputs or unhappy customers with your existing application

Ready to lead your team in building a data flywheel so you can leverage feedback

This Course Is For You If You Are

By the end of this course, participants will be able to

Implement a systematic approach to developing and improving RAG applications using the Data and Evals Flywheel methodology.

Design and execute fast, unit test-like evaluations to assess retrieval capabilities, including precision and recall metrics.

Generate and utilize synthetic data for rapid evaluation and iteration of RAG systems.

Apply fine-tuning strategies for embedding models and implement hard negative mining techniques to enhance search relevance.

Classify different types of queries and conduct bottleneck analysis to identify performance limitations in RAG systems.

Differentiate between limited inventory and limited capabilities issues, and develop strategies to address both.

Design and implement specialized indices for various data types, including documents, images, tables, and SQL databases.

Apply synthetic text chunk generation and summarization techniques to improve retrieval performance across different modalities.

Develop efficient query routing systems and implement effective index fusion strategies for complex RAG setups.

Evaluate the performance of both query routing and individual indices separately to optimize overall system performance.

Design and integrate both explicit and implicit feedback mechanisms to drive continuous system improvement.

Follow a repeatable process to continually evaluate and improve your RAG application

Jason Liu

Through multiple real examples of well-run experiments and real stories at Microsoft, Amazon, and Airbnb, you will see the humbling reality that we are terrible at assessing the values of ideas. Trivial changes can be surprisingly useful, whereas large efforts often fail. Accelerate innovation by building Minimum Viable Products and Features (MVPs) and make the organization evidence-based and humbler, as it adopts and learns to use evidence from the gold standard in science: the controlled experiment. You will understand the challenges in designing and running trustworthy controlled experiments, or A/B tests, including the importance of the Overall Evaluation Criterion (OEC), scaling, pitfalls, and Twyman's law. The 2nd week covers additional topics, some more technical, including cultural challenges, institutional memory, maturity model, observational causal studies, offline evaluations, AI/Machine learning and triggering, Bayesian vs. Frequentist, scaling, build vs. buy, challenges, and requested topics.

Empower your organization to be data-driven and innovative

<ul><li>Introduction to controlled experiments, or A/B testing</li><li>Interesting experiments – you’re the decision maker</li><li>Organizational tenets</li><li>Advantages and limitations of controlled experiments</li></ul>

Introduction

<ul><li>Interesting example</li><li>End-to-end experiment: Running and Analyzing an Experiment</li><li>Metrics</li><li>The OEC</li></ul>

End-to-end example, and Metrics and the OEC

<ul><li>Interesting example</li><li>P-values and statistical power</li><li>End-to-end experiment: speed matters</li><li>Twyman’s Law and Trustworthy Experimentation</li><li>Generating ideas, prioritization, ethics</li></ul>

P-values and power, End-to-end Example 2, Twyman's law and Trustworthy Experimentation

<ul><li>Interesting example</li><li>Cultural challenge, Institutional memory, maturity model</li><li>Observational causal studies and quasi-experimental designs</li><li>Pitfalls in observational studies</li></ul>

Cultural challenges, Prioritization, AI/Machine Learning, Complementary Techniques and Observational Causal Studies

<ul><li>AI/Machine Learning model, and triggering</li><li>Leakage and interference</li><li>Bayesian A/B Testing</li><li>Experimentation Platform and scaling</li><li>Build vs. Buy</li><li>Challenges</li><li>Requested Topics</li></ul>

Advanced topics, challenges, and requested topics

Three x 2-hour sessions in week 1 on Monday, Tuesday, and Thursday

Week 1: Mon, Tue, Thu

Two x 2-hour sessions in week 2 on Monday and Thursday

Week 2: Mon, Thu

Optional Q&A

Data science managers and scientists will be able to design and interpret the experiment results in a trustworthy manner

Program managers focused on growth, revenue, conversions, and prioritization will understand how to provide the org with robust clear metric

Engineering leaders will be able to make the organizations more data-driven and efficient with fewer severe incidents through A/B tests

Who is this class for?

What you will learn from this 10-hour course

Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business