Building software, breaking things, and writing down what I learned.


Tag: artificial-intelligence

  • The AI Wasn’t the Hard Part

    The AI Wasn’t the Hard Part

    Building an AI-powered feature, navigating frontend conventions, and wrestling with a codebase that wouldn’t sit still.

    I recently worked on an AI-powered feature that takes an insertion order (IO) PDF, extracts campaign information, and uses it to populate a draft in our DSP campaign builder.

    Sounds like the AI would be the complicated part, right?

    Not so much.

    I’d done some prompt engineering before, so I had a decent starting point. The MCP tools I needed were already built, although a few API adjustments were necessary to get the right data. Connecting the agent to our internal assistant infrastructure was new territory for me, but other engineers in my pod had done similar work. I had existing implementations to reference and people I could turn to when I needed help.

    Then I got to the frontend.

    Between navigating component conventions, managing asynchronous processing and form state, and discovering that another feature flag was about to rearrange the campaign builder’s fields and step order, I had plenty to keep me occupied.

    Don’t get me wrong. There was still engineering involved in the AI integration. But the challenges that took the most effort weren’t really AI problems at all.

    Apparently, adding AI to an application doesn’t exempt you from the usual software engineering headaches.

    Getting the Data Where It Belongs

    Before diving into implementation, I worked through the requirements and architecture using internal AI-assisted planning tools. I reviewed the proposed approach and documentation before getting feedback from my product manager and engineering manager.

    The basic workflow sounded simple enough: upload a PDF, extract the relevant campaign information, and populate the appropriate fields.

    Except there were a few catches.

    For starters, extraction wasn’t synchronous. Once a user uploaded a PDF, the browser extracted its text and sent it for processing. The frontend then needed to wait for the assistant to finish and retrieve the results.

    That meant managing loading states, polling for responses, handling timeouts, and accounting for users who might upload another document before the first one finished processing.

    Consider what happens when someone uploads one IO, changes their mind, and uploads another before the first extraction completes.

    The application can’t just accept whichever response comes back last. Otherwise, you risk populating the campaign with information from a document the user has already replaced.

    In my implementation, a new upload stops polling for the previous one, preventing its results from being applied to the current draft. Failed or timed-out extractions also need to be communicated to the user rather than leaving the interface stuck in a loading state.

    Then there was the question of what to do with the extracted data.

    The campaign builder was already a multi-step form with its own validation and save behavior. I couldn’t just take the extracted values, shove them into the form, and hope for the best.

    The user needed to review and approve the extracted information before it was applied. Even then, not every value belonged in the form immediately.

    Some fields could be populated right away, while others needed to be staged until the user reached the appropriate step.

    Re-uploading a document introduced another complication. What if the user had already reviewed or modified part of the campaign? The new extraction couldn’t simply overwrite everything.

    That meant distinguishing between information that could be replaced and information that needed to be preserved, while making sure failed operations didn’t leave the form in an inconsistent state.

    Cursor helped considerably with the implementation, particularly when working through unfamiliar parts of the codebase and some of these state management challenges. But there was still plenty of reviewing, testing, and figuring out how everything needed to fit together.

    Because apparently extracting the data was only half the battle. Getting it into the right place without breaking something else was where things got interesting.

    The Front End Had Other Plans

    I’ve worked with Figma designs plenty of times, so I wasn’t expecting the UI to be my biggest headache.

    That was optimistic.

    The issue wasn’t necessarily that the designs were bad. It was that they left a lot of questions unanswered once I started translating them into actual components.

    Which components were intended? Should I use something straight out of Ant Design or one of our internally customized versions? What were the conventions for styling and handling different processing states?

    And seriously, what color was I supposed to use?

    There were also existing frontend patterns to consider. I wasn’t building a standalone application where I could make every decision from scratch. This feature needed to look and behave like it belonged in the campaign builder.

    I spent a fair amount of time working through those questions with the frontend SME in my pod. We compared options against existing conventions, figured out how to handle ambiguous components and statuses, and brought in the design team when we needed additional clarification.

    It wasn’t particularly glamorous work, but it mattered. The difference between a feature that technically works and one that feels like a natural part of an application often comes down to those little decisions.

    And apparently, figuring out which alert color to use can take more effort than connecting an AI agent to an API.

    Feature Flags: Because One Wasn’t Enough

    Just when I thought I had a pretty good handle on the implementation, I learned about another feature flag that was going to change the campaign builder’s fields and the order of its steps.

    Information that would have been really useful a little earlier.

    I’d already built a substantial portion of the ingestion workflow around the existing configuration. Now I needed to account for an updated version of the same application.

    Fortunately, I didn’t have to start over.

    The ingestion feature had its own feature flag, and the campaign builder changes were controlled by a separate one.

    Rather than tie my implementation to the upcoming changes, I kept what I’d built and made adjustments so the ingestion workflow could handle either version of the campaign builder.

    That meant accounting for differences in the available fields and where extracted information needed to be populated, depending on which configuration was active.

    Was it annoying? Absolutely.

    But it also reinforced something important about working in a shared codebase: you’re rarely developing against a static application.

    Other features are being built, existing functionality is changing, and sometimes those changes overlap with your own work in ways you didn’t anticipate.

    You can plan for the requirements you know about. The trick is being able to adapt when you discover the ones you didn’t.

    So, What Was Actually Hard?

    Looking back, the most challenging parts of this project weren’t the ones I initially associated with building an AI agent.

    The AI integrations came together relatively smoothly because I had existing tools, examples, and teammates to lean on.

    The bigger challenges were fitting the extraction workflow into an established application, managing state without disrupting existing behavior, navigating frontend conventions, and adapting to changes I hadn’t known were coming.

    None of that is particularly groundbreaking. It’s just software engineering.

    And I think that’s my biggest takeaway from this project.

    For all the excitement surrounding AI development, the fundamentals haven’t really changed. You still need to understand the systems you’re working with, account for things going wrong, and occasionally ask someone who knows a particular corner of the codebase better than you do.

    AI-assisted tools can make development faster, but they don’t make those decisions for you. At least, not decisions you should blindly trust.

    I went into this project expecting to spend most of my time figuring out the AI components. Instead, I spent a surprising amount of it dealing with frontend conventions, existing application behavior, and feature flags.

    Would I do it again? Absolutely.

    Would I ask a few more questions about the frontend before getting started?

    Also absolutely.

  • Optimizing Task Scheduling in Task Alchemist: A Monthly Recap

    Optimizing Task Scheduling in Task Alchemist: A Monthly Recap

    This month, my focus for Task Alchemist was to enhance the task scheduling functionality by integrating AI-driven optimization techniques and ensuring seamless user experiences with Google Calendar imports. Additionally, significant time was spent debugging task scheduling issues and utilizing Python for various backend processes. This post chronicles my journey, highlighting the features developed, challenges faced, and lessons learned.

    New Features Developed

    AI-Driven Task Scheduling

    Goal: Implement reinforcement learning and regression-based scheduling to optimize task assignments.

    Implementation:

    1. Reinforcement Learning Integration:
      • Training Model on Schedule: The training of the reinforcement learning model happens on a schedule using a Cloud Run container. The model processes real user task data stored in Firebase.
      • Optimized Scheduling: The optimized task scheduling can be triggered from the React app whenever the user pushes the “Schedule Tasks” button or drags a task from the inbox to the calendar.
      • Database Updates: Upon scheduling, tasks are marked as scheduled, and corresponding appointments are saved in the database.

    Google Calendar Integration

    Goal: Ensure seamless import and persistence of Google Calendar events into FullCalendar.

    Implementation:

    1. Data Import:
      • Implemented a feature to import events from Google Calendar, tagging each imported event with source: 'google'.
    2. Persistence:
      • Used Firebase to store user-selected calendars, ensuring that these selections persist across sessions.
    3. Task Scheduling Around Events:
      • Adjusted Firebase Cloud Functions to calculate priority weighting and schedule tasks around imported Google Calendar events.

    Debugging Task Scheduling

    Challenges: This month, a significant amount of time was spent debugging issues related to task scheduling. The primary challenges included:

    • Data Inconsistencies: Ensuring data consistency between the tasks scheduled in the application and those stored in Firebase.
    • Edge Cases: Handling edge cases such as overlapping tasks and tasks with conflicting priorities.

    Approach:

    • Logging and Monitoring: Implemented extensive logging and monitoring to trace issues and understand the flow of data.
    • Unit Testing: Developed unit tests to validate the correctness of the scheduling logic.
    • Python Scripts: Utilized Python for backend processes, including data cleaning and preprocessing before feeding it into the reinforcement learning model.

    Retrospective

    What Went Right

    • Successfully integrated reinforcement learning for task scheduling, significantly improving the efficiency and accuracy of task assignments.
    • Streamlined the process of importing Google Calendar events, providing users with a unified view of their schedules.
    • Achieved a high level of persistence for user preferences, enhancing the overall user experience.
    • Effective debugging of task scheduling issues, ensuring data consistency and handling edge cases.

    What Went Wrong

    • Encountered challenges with the initial setup of Cloud Run for model training, leading to delays.
    • Faced difficulties in handling edge cases with Google Calendar events, such as recurring events and time zone differences.

    Areas for Improvement

    • Time Management: Allocate more time for testing and debugging complex integrations.
    • User Feedback: Conduct more frequent user testing to identify and address issues early.
    • Documentation: Improve documentation for new features to assist future development and onboarding of new team members.

    Conclusion

    This month has been a period of significant growth and learning. The advancements in AI-driven task scheduling and Google Calendar integration, coupled with extensive debugging efforts, have brought us closer to our vision for Task Alchemist. Moving forward, I aim to refine these features further, focusing on user feedback and continuous improvement.

    By leveraging the knowledge gained from previous courses and experience, I am confident in tackling the next phase of development. The journey continues, and I look forward to the exciting challenges ahead.