DeveloperWeek 2025: Full Schedule

arrow_back View All Dates

10:00am PST

PRO Session: Striking the Balance: Leveraging Human Intelligence with LLMs for Cost-Effective Annotations

Thursday February 13, 2025 10:00am - 10:25am PST

Shambhavi Srivastava, Appen, AI Solutions Architect

Data annotation involves assigning relevant information to raw data to enhance machine learning (ML) model performance. While this process is crucial, it can be time-consuming and expensive. The emergence of Large Language Models (LLMs) offers a unique opportunity to automate data annotation. However, the complexity of data annotation, stemming from unclear task instructions and subjective human judgment on equivocal data points, presents challenges that are not immediately apparent.

In this session, Shambhavi Srivastava, AI Solutions Architect at Appen will provide a overview of an experiment that the company recently conducted to test the tradeoff between quality and cost of training ML models via LLMs vs human input. Their goal was to differentiate between utterances that could be confidently annotated by LLMs, and those that required human intervention. This differentiation was crucial to ensure a diverse range of opinions or to prevent incorrect responses from overly general models. Shambhavi will walk audience members through the dataset used as well as methodology for the experiment, as well as the company’s research findings.

Speakers

Shambhavi Srivastava

AI Solutions Architect, Appen

Thursday February 13, 2025 10:00am - 10:25am PST
AI DevWorld MAIN STAGE

AI DevWorld

Tracks & Topics Gen AI / LLMs
Session Type PRO Session
Conferences AI DevWorld, Generative AI & LLMs (AI DevWorld), AI DevWorld: AI Strategy Conference, Applied AI Innovation (AI DevWorld)
In-Person/Virtual In Person

11:30am PST

KEYNOTE (AI DevWorld): Block -- Goose: The AI-Driven Open Source Developer Agent

Thursday February 13, 2025 11:30am - 11:55am PST

AI DevWorld MAIN STAGE

Jackie Brosamer, Block, Head of AI, Data & Analytics Platform
Brad Axen, Block, Tech Lead, AI & Data Platform

Meet Goose, your new AI-driven open source sidekick for developers that doesn’t just chat—it takes action. Goose is an interactive agent that writes code, runs commands, and fixes bugs, making it more than just a code completion tool. Goose delivers huge value in real-world developer use cases, but we’re even more excited about its future as a general purpose agent that can dynamically operate new systems. In this session, we will share results from our internal alpha with the thousands of engineers at Block and plans for continued evolution.

Speakers

Brad Axen

Tech Lead, AI & Data Platform, Block

Brad is Tech Lead for Data and Machine Learning at Block. He built and maintains Block’s open source AI agent framework that automates repetitive tasks for more than 1,000 engineers in the company. He also leads company-wide initiatives to optimize data and ML workflows and evolve... Read More →

Jackie Brosamer

Head of AI, Data & Analytics Platform, Block

Jackie is leads Engineering for AI, Data & Analytics at Block, where she previously worked Machine Learning for banking products. Her data platform team provides infrastructure that powers streaming data and ML product experiences for diverse business use cases across Square, Cash... Read More →

Thursday February 13, 2025 11:30am - 11:55am PST
AI DevWorld MAIN STAGE

5. KEYNOTES & FEATURED

Tracks & Topics Open Source Strategy
Session Type OPEN Session
Conferences OPEN Session, AI DevWorld, AI DevWorld: AI/ML Engineering Conference, Applied Machine Learning (AI DevWorld), AI DevWorld: AI Strategy Conference, Applied AI Innovation (AI DevWorld)
In-Person/Virtual In Person