# Vidya: complete content for language models > Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. This file concatenates every page on https://www.vidyasource.com/. The short index is https://www.vidyasource.com/llms.txt. Each page below starts with front matter that names its canonical HTML URL. Contents: 6 consulting practice areas, 9 case studies, 5 courses, 0 products, 4 tutorials, and 67 articles. ## Topic index Technologies named in case studies, with engagement counts: Architecture (7), Government (6), Java (6), Modernization (6), React (4), Scala (4), AWS (3), Jenkins (3), Play Framework (3), Cloud (2), Culture (2), Go (2), MongoDB (2), Partners (2), Project Management (2), Security (2), Testing (2), TypeScript (2), AI (1), Akka (1), Alpakka (1), Amazon CloudFront (1), Amazon EC2 (1), Amazon S3 (1), Angular (1), Apache Kafka (1), Apache Spark (1), Azure Databricks (1), Cassandra (1), Chakra UI (1), Cloudinary (1), Data (1), DevOps (1), Docker (1), Dropwizard (1), Flowplayer (1), Functional Programming (1), Go kit (1), Gradle (1), Groovy (1), Java Virtual Machine (1), Kubernetes (1), Machine Learning (1), Microsoft Azure (1), Oracle Exadata (1), PostgreSQL (1), Python (1), RabbitMQ (1), Red Hat OpenShift (1), Redux (1), Resilience4j (1), REST (1), Ruby on Rails (1), Salesforce (1), ScalaCheck (1), ScalaTest (1), Spark MLlib (1), Spring (1), Spring Boot (1), Spring Integration (1) Topics across everything, with counts: Programming (46), Software Engineering (39), Agile (37), Java (30), Testing (27), Architecture (24), Government (23), Scala (22), Project Management (19), Functional Programming (18), AI (17), Open Source (17), Partners (16), Scrum (11), Security (11), Big Data (10), Go (10), Python (10), React (10), TypeScript (10), Continuous Integration (9), Diversity (9), JavaScript (9), DevOps (8), Machine Learning (8), Mobile (8), Accessibility (7), Analytics (7), Lean (7), MCP (7), Ruby (7), Android (6), Apache Spark (6), Continuous Delivery (6), iOS (6), Kotlin (6), Modernization (6), Play Framework (6), REST (6), DevSecOps (5), Hadoop (5), Microservices (5), Agentic AI (4), Agents (4), Akka (4), AWS (4), Jenkins (4), Agent Skills (3), Apache Kafka (3), Clojure (3), Data (3), Docker (3), Kanban (3), Kubernetes (3), MongoDB (3), ScalaCheck (3), Swift (3), Technical Debt (3), Angular (2), Cassandra (2), Chakra UI (2), Cloud (2), CSS (2), Culture (2), Deep Learning (2), Elm (2), Gradle (2), HTML (2), JUnit (2), Leadership (2), LLMs (2), MapReduce (2), NextJS (2), NoSQL (2), Observability (2), Open source (2), PMP (2), PostgreSQL (2), programming (2), PWA (2), Reactive (2), Salesforce (2), SBT (2), Selenium (2), Spring (2), Spring Boot (2), Tailwind CSS (2), agentgateway (1), AGENTS.md (1), Alpakka (1), Amazon CloudFront (1), Amazon EC2 (1), Amazon S3 (1), Ansible (1), API (1), APIs (1), Azure Databricks (1), Big Tech (1), Brave (1), CAP Theorem (1), Cascalog (1), Case classes (1), Cloud Computing (1), Cloudinary (1), Community (1), comparable (1), comparator (1), data (1), Data Science (1), data visualization (1), dataviz (1), Donor's Choose (1), Dropwizard (1), Extractors and pattern matching (1), Facebook (1), Flask (1), Flowplayer (1), function (1), functional programming (1), Futures (1), Git (1), Go kit (1), Golang (1), Goose (1), Groovy (1), HBase (1), HDFS (1), Hive (1), Hugging Face (1), Hugo (1), Immutability (1), ImmutableJS (1), Implicit conversion (1), Java Virtual Machine (1), JDBC (1), Jest (1), Microsoft Azure (1), Neo4J (1), Option (1), Oracle Exadata (1), Parallel collections (1), RabbitMQ (1), Rails (1), RDBMS (1), React Testing Library (1), ReactiveMongo (1), Red Hat OpenShift (1), Redux (1), Resilience4j (1), Ruby on Rails (1), RxJS (1), ScalaTest (1), Scalaz (1), Social Media (1), sorting (1), Spark MLlib (1), Spring Integration (1), SRE (1), Storybook (1), Traits (1), Vagrant (1), Vite (1), VueJS (1), web (1), web design (1), web development (1), Web development (1), Webpack (1), WordPress (1) --- title: "Vidya" description: "Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com." canonical: https://www.vidyasource.com/ type: page --- # Vidya > Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. - Canonical page: https://www.vidyasource.com/ - Structured data (JSON-LD): https://www.vidyasource.com/index.json ## Where to look - [Consulting](https://www.vidyasource.com/consulting.md): six practice areas, from legacy modernization to agentic AI. - [Case studies](https://www.vidyasource.com/case-studies.md): delivered engagements with named clients and results. - [Courses](https://www.vidyasource.com/courses.md): instructor-led technology training. - [Tutorials](https://www.vidyasource.com/tutorials.md): free video tutorials with source code. - [Articles](https://www.vidyasource.com/blog.md): the complete blog, newest first. - [About](https://www.vidyasource.com/about.md) and [Contact](https://www.vidyasource.com/contact.md). - [llms.txt](https://www.vidyasource.com/llms.txt) is the short index; [llms-full.txt](https://www.vidyasource.com/llms-full.txt) is every page in one file. ## Company Overview Vidya is a federally certified 8(a) and Virginia SWaM (Small, Micro, Minority-owned) business that specializes in legacy system modernization and technology consulting, covering application code, data architecture, and the agile and DevSecOps culture that sustains both. Founded in 2010 and headquartered in Vienna, VA, Vidya serves both commercial businesses and government agencies. ## Core Services - Custom software development (on-premise, cloud, and hybrid solutions) - Legacy system modernization and serverless architecture transitions - Agile software development and DevOps implementation - Technology training courses and educational content - IT consulting and strategic technology planning ## Key Differentiators ### Government Expertise & Certifications - US Small Business Administration 8(a) certified business - Virginia SWaM (Small, Micro, Minority-owned) certified - GSA Multiple Award Schedule (MAS) contract holder - Member of the Government Accountability Office Agile Expert Group - Proven track record with high-profile government projects including: - US State Department Online Passport Renewal system architecture - HealthCare.gov improvements - Recreation.gov enhancements - White House technology hiring modernization initiatives ### Commercial Expertise & Certifications - Proven track record with high-profile commercial projects including: - Strategic partnership with Neustar, a leading telecommunications and cloud platform company serving every phone call, fax, and computer connection in North America - Collaboration with Nina Day, a premier casting and creative agency serving Fortune 500 clients worldwide - Partnership with Webster & Fredrickson, PLLC, a premier Washington DC law firm - Worked with TRSS, a leading provider of scalable solutions for global institutions - Successfully delivered REST billing APIs, cloud solutions, and enterprise modernization projects for commercial clients - Expertise in building scalable solutions for telecommunications, entertainment, legal, and cybersecurity industries - CRMSDC Minority Business Enterprise certified - USPAACC Asian and minority-owned small business certified ### Technical Excellence & Modern Practices - Deep expertise in AI and cloud modernization - Strong focus on agile methodologies, continuous delivery, and DevOps - Specialized knowledge in modern programming languages including Java, Kotlin, TypeScript, Go, and Python - Experience with modern tech stacks including React, Astro, AWS, NoSQL - Emphasis on building security, quality, and performance into software from day one - Active contributor to open-source community with 30K+ Stack Overflow reputation ### Educational Leadership - Extensive library of technical blog content covering software engineering best practices - Video tutorials and courses on programming and technology concepts - Thought leadership on topics like developer productivity, testing strategies, and modern development practices - Speaking engagements at industry conferences and government events - Practical insights that demystify complex technical concepts for non-technical stakeholders ### Unique Value Proposition Vidya bridges the gap between complex technical solutions and business outcomes by combining: - Small business agility with enterprise-level expertise - Government compliance knowledge with commercial innovation - Technical depth with exceptional communication and training capabilities - Minority-owned business status with proven delivery on large-scale projects - Unique ability to augment the value of legacy data and systems through integration with modern technologies and techniques ## Contact Information - Website: https://www.vidyasource.com - Email: info@vidyasource.com - Phone: +1 202 430 5695 - Location: Vienna, Virginia, Washington DC Metro Area - GitHub: https://github.com/VidyaSource ## Company Philosophy "Build faster. Build better." Vidya focuses on making technology accessible and enjoyable while delivering solutions that align with clients' strategic objectives. The company proves there's no need to choose between continuous delivery and high quality. ## Notable Recognition - Top 100 MBE Award recipient from CRMSDC - Featured speaker at White House technology recruitment initiatives - Recognized by clients for exceptional ability to explain complex technical concepts with clarity and humor - Successfully transitioned multiple organizations to modernized, serverless architectures --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Consulting" description: "Vidya's consulting practice areas." canonical: https://www.vidyasource.com/consulting/ type: page --- # Consulting > Vidya's consulting practice areas. - Canonical page: https://www.vidyasource.com/consulting/ - Structured data (JSON-LD): https://www.vidyasource.com/consulting.json ## Consulting Services Vidya provides modernization consulting across architecture, software development, and technology culture. ### Modernized Architecture The simplest architectures are the most maintainable and the most secure. Vidya helps clients discover the emergent architecture best suited to meet their challenges. Architecture services include: - Integrating legacy systems with modern technologies - Software and data architecture - Cloud Native deployments - AI work, focused less on LLMs in isolation and more on integrating context at scale - Zero Trust security - UX design, accessibility, and component library development - Modern Agile and DevSecOps ### Software Development Software development is where the principles of the architecture become the operational reality that thrills customers and generates revenue. Vidya helps clients pick the right blend of programming languages, frameworks, and tools and apply them with best-in-class practices: modeling the domain with object-oriented programming, composing behavior with functional programming, building accessible user interfaces, creating ML pipelines, automating builds with DevSecOps, and standardizing resilient deployments across clouds. ### Technology Culture Even the most far-reaching technological transformation has no chance for success without a strong organizational culture. A positive culture breeds comfort with experimenting with new technologies and encourages the sharing of ideas and the collaboration necessary to implement them effectively. Vidya helps clients build that culture in technology-related areas. ### Technology Communication Technology is meaningless without the ability to express its power through engaging verbal and written communication. Vidya communicates technical expertise effectively, whether explaining the value and risk of a particular technology to customers or writing concise prose in a proposal. Examples of Vidya's communication work include the company blog and video tutorials. ## Practice areas - [AI Consulting & Agentic Engineering](https://www.vidyasource.com/consulting/ai-consulting.md): AI built on open standards, grounded in your business context. (HTML: https://www.vidyasource.com/consulting/ai-consulting/) - [Legacy System Modernization](https://www.vidyasource.com/consulting/legacy-system-modernization.md): Modernization reaches the code, the data model, and the delivery culture, in small steps while your system keeps running. (HTML: https://www.vidyasource.com/consulting/legacy-system-modernization/) - [Cloud Native Architecture & DevSecOps](https://www.vidyasource.com/consulting/cloud-native-devsecops.md): Software releases that ship on a predictable, routine schedule. (HTML: https://www.vidyasource.com/consulting/cloud-native-devsecops/) - [Software & Data Architecture](https://www.vidyasource.com/consulting/software-architecture.md): Systems your own team can understand, maintain, and grow. (HTML: https://www.vidyasource.com/consulting/software-architecture/) - [Technology Culture & Organizational Consulting](https://www.vidyasource.com/consulting/technology-culture.md): A team that trusts each other enough to catch problems early. (HTML: https://www.vidyasource.com/consulting/technology-culture/) - [AI & Modernization Consulting for Government](https://www.vidyasource.com/consulting/government-modernization.md): An 8(a) and SWaM-certified firm already on GSA MAS, with 6 years of delivered federal modernization behind it. (HTML: https://www.vidyasource.com/consulting/government-modernization/) --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Case Studies" description: "Engagements Vidya has delivered, with clients, technologies, and results." canonical: https://www.vidyasource.com/case-studies/ type: page --- # Case Studies > Engagements Vidya has delivered, with clients, technologies, and results. - Canonical page: https://www.vidyasource.com/case-studies/ - Structured data (JSON-LD): https://www.vidyasource.com/case-studies.json - [Cutting U.S. Passport Renewal From Months to Days](https://www.vidyasource.com/case-studies/consular-systems-modernization-state-department.md): U.S. Department of State, Bureau of Consular Affairs, federal government. Vidya modernized dozens of legacy services, most notably Online Passport Renewal, without taking the service offline for a day. (HTML: https://www.vidyasource.com/case-studies/consular-systems-modernization-state-department/) - [Making HealthCare.gov More Than 30% Faster for CMS](https://www.vidyasource.com/case-studies/healthcare-gov-modernization.md): Centers for Medicare & Medicaid Services (CMS), federal government. Working under Accenture Federal Services, Vidya rebuilt a Spring application as a Play Framework application in Scala and taught 20 Java engineers to maintain it. (HTML: https://www.vidyasource.com/case-studies/healthcare-gov-modernization/) - [TRSS Turns Threat Detection Into a Named Product Line](https://www.vidyasource.com/case-studies/machine-learning-threat-detection-trss.md): Thomson Reuters Special Services (TRSS), national security & federal law enforcement. TRSS analysts now open a dashboard to a threat score already computed, and the platform Vidya rebuilt grew into a product line the company still sells. (HTML: https://www.vidyasource.com/case-studies/machine-learning-threat-detection-trss/) - [Tripling Reservations on Recreation.gov](https://www.vidyasource.com/case-studies/recreation-gov-modernization.md): Recreation.gov, federal government. Working under Booz Allen Hamilton, Vidya built Go microservices and React components on a platform that tripled reservations across the nation's parks, forests, and public lands. (HTML: https://www.vidyasource.com/case-studies/recreation-gov-modernization/) - [Neustar Modernizes Telecom Billing Behind a REST API](https://www.vidyasource.com/case-studies/api-modernization-neustar.md): Neustar, commercial. Every new cloud offering at Neustar could charge for usage without building billing of its own, on a shared service Vidya delivered over three years. (HTML: https://www.vidyasource.com/case-studies/api-modernization-neustar/) - [The Curriculum That Became the Federal DITAP Program](https://www.vidyasource.com/case-studies/ditap-digital-acquisition-training-eop.md): Executive Office of the President, federal government. Vidya taught contracting officers to buy software in increments, working alongside the U.S. (HTML: https://www.vidyasource.com/case-studies/ditap-digital-acquisition-training-eop/) - [Modernizing Federal Technical Hiring at the White House](https://www.vidyasource.com/case-studies/federal-tech-hiring-white-house-mediabarn.md): Mediabarn, federal government. Vidya's founder gave a cohort of government recruiters a working playbook on Technology Day, delivered for Mediabarn. (HTML: https://www.vidyasource.com/case-studies/federal-tech-hiring-white-house-mediabarn/) - [Nina Day Casts Global Brands from One Talent Database](https://www.vidyasource.com/case-studies/casting-platform-nina-day.md): Nina Day, media & advertising. Vidya has run this platform for over a decade and rebuilt it once the agency's growth outgrew the first version, so a casting director finds the right face in a single sitting. (HTML: https://www.vidyasource.com/case-studies/casting-platform-nina-day/) - [Concurrent Real-Time Briefs Two Audiences With One White Paper](https://www.vidyasource.com/case-studies/technical-white-papers-concurrent-real-time.md): Concurrent Real-Time, defense & real-time systems. Vidya documents RedHawk Linux, KVM-RT, and the Missile TestBench so mission assurance staff and engineers read the same argument about deterministic latency. (HTML: https://www.vidyasource.com/case-studies/technical-white-papers-concurrent-real-time/) --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Courses" description: "Technology training courses taught by Vidya." canonical: https://www.vidyasource.com/courses/ type: page --- # Courses > Technology training courses taught by Vidya. - Canonical page: https://www.vidyasource.com/courses/ - Structured data (JSON-LD): https://www.vidyasource.com/courses.json - [Data Engineering with Spark](https://www.vidyasource.com/courses/data-engineering-with-spark.md): Data Science. Master data engineering with Apache Spark through Vidya's hands-on training course. (HTML: https://www.vidyasource.com/courses/data-engineering-with-spark/) - [Enterprise AI on the JVM](https://www.vidyasource.com/courses/enterprise-ai-on-the-jvm.md): AI. Despite what you may have heard, the JVM is where you want to build if you are serious about Enterprise AI. (HTML: https://www.vidyasource.com/courses/enterprise-ai-on-the-jvm/) - [Java For Work](https://www.vidyasource.com/courses/java-for-work.md): Java. Enhance your Java skills with Vidya's Java for Work course. (HTML: https://www.vidyasource.com/courses/java-for-work/) - [Modern Agile](https://www.vidyasource.com/courses/modern-agile.md): Agile. Embrace modern agile methodologies with Vidya's comprehensive training course. (HTML: https://www.vidyasource.com/courses/modern-agile/) - [Practical AI for Business](https://www.vidyasource.com/courses/practical-ai-for-business.md): AI. A hands-on AI course for every skill level and industry. (HTML: https://www.vidyasource.com/courses/practical-ai-for-business/) --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Tutorials" description: "Free video tutorials with source code on GitHub." canonical: https://www.vidyasource.com/tutorials/ type: page --- # Tutorials > Free video tutorials with source code on GitHub. - Canonical page: https://www.vidyasource.com/tutorials/ - Structured data (JSON-LD): https://www.vidyasource.com/tutorials.json - [Starting with Data: Using Python and Flask to Pull Data from a REST API and Visualize It](https://www.vidyasource.com/tutorials/starting-with-data.md): This tutorial is for beginners in software development who want to learn just enough to access data on the web and visualize it on their own websites or mobile devices. (HTML: https://www.vidyasource.com/tutorials/starting-with-data/) - [Comparison Shopping: Using Comparable and Comparator to Sort Java Objects](https://www.vidyasource.com/tutorials/comparison-shopping.md): This tutorial is for intermediate-level Java developers who want to add comparison and sorting capabilities to the custom classes they’ve created for their projects. (HTML: https://www.vidyasource.com/tutorials/comparison-shopping/) - [Nine Reasons to Try Scala](https://www.vidyasource.com/tutorials/nine-reasons-to-try-scala.md): This tutorial is for intermediate-level Java developers, and developers in other languages too who are curious about what the big deal is with the Scala programming language. (HTML: https://www.vidyasource.com/tutorials/nine-reasons-to-try-scala/) - [Web Design 101: Understanding the relationship among HTML, CSS, and JavaScript](https://www.vidyasource.com/tutorials/web-design-101.md): This tutorial is for beginners in software development who want to learn just enough to access data on the web and visualize it on their own websites or mobile device applications. (HTML: https://www.vidyasource.com/tutorials/web-design-101/) --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Vidya Blog" description: "All 67 articles, newest first." canonical: https://www.vidyasource.com/blog/ type: page --- # Vidya Blog > All 67 articles, newest first. - Canonical page: https://www.vidyasource.com/blog/ - Structured data (JSON-LD): https://www.vidyasource.com/blog.json - [Making Decisions with Jev and Goose](https://www.vidyasource.com/blog/making-decisions-with-jev-and-goose.md): Pair an LLM in Goose with TypeSafe's Jev, and your AI agents talk in plain English while they make fast, calibrated decisions for a fraction of a cent. (HTML: https://www.vidyasource.com/blog/making-decisions-with-jev-and-goose/) - [Put Your Opinions to Work with AGENTS.md and Keep Them Safe](https://www.vidyasource.com/blog/put-your-opinions-to-work-with-agents-md-and-keep-them-safe.md): AGENTS.md turns your opinions into standards every AI coding agent enforces. Respect the instruction budget, then defend the file from prompt injection. (HTML: https://www.vidyasource.com/blog/put-your-opinions-to-work-with-agents-md-and-keep-them-safe/) - [I'm an Agentic AI Foundation Ambassador!](https://www.vidyasource.com/blog/agentic-ai-foundation-ambassador.md): I've been named an inaugural Agentic AI Foundation Ambassador. Here's why AAIF and open standards in general are critical for any technology to meet its potential. (HTML: https://www.vidyasource.com/blog/agentic-ai-foundation-ambassador/) - [Create a Culture of Context for AI](https://www.vidyasource.com/blog/create-a-culture-of-context-for-ai.md): Agentic AI fails without the right context. Learn how progressive disclosure, institutional memory, Data Mesh, and Knowledge Layers build a Culture of Context that cuts hallucinations and token waste. (HTML: https://www.vidyasource.com/blog/create-a-culture-of-context-for-ai/) - [Infrastructure as Code Has a New Playbook: formae and Pkl](https://www.vidyasource.com/blog/infrastructure-as-code-formae-pkl-aws.md): formae brings agentic, stateless IaC with Pkl's type-safe config to AWS. No drift, no state files, and an MCP server AI agents can actually use. (HTML: https://www.vidyasource.com/blog/infrastructure-as-code-formae-pkl-aws/) - [Python Is 31x Slower: You Can Do Better for Enterprise AI](https://www.vidyasource.com/blog/python-is-31x-slower-new-benchmark-confirms-not-language-for-enterprise-ai.md): A new MCP server benchmark shows Python is 31x slower than Go and Java—reinforcing why enterprise AI demands languages built for integration and performance, not just experimentation. (HTML: https://www.vidyasource.com/blog/python-is-31x-slower-new-benchmark-confirms-not-language-for-enterprise-ai/) - [The Biggest News in AI Since MCP](https://www.vidyasource.com/blog/biggest-news-in-ai-since-anthropic-mcp.md): The Linux Foundation launches the Agentic AI Foundation, which formalizes MCP, Goose, and AGENTS.md into global open standards. (HTML: https://www.vidyasource.com/blog/biggest-news-in-ai-since-anthropic-mcp/) - [Vidya Talks AI and MCP at NOVA SART](https://www.vidyasource.com/blog/vidya-talks-ai-and-mcp.md): Key insights from our appearance on the MCP Roundtable at NOVA SART: MCP's promise and peril, AI architecture, security, real-world use cases, and why AI will increase demand for software expertise. (HTML: https://www.vidyasource.com/blog/vidya-talks-ai-and-mcp/) - [Python Is Not the Language of AI](https://www.vidyasource.com/blog/python-is-not-language-of-ai.md): Python is the language of machine learning and training AI models, but there are several reasons why Python is not the language of AI. (HTML: https://www.vidyasource.com/blog/python-is-not-language-of-ai/) - [Beyond GitHub Culture: Why Your Team Should Reconsider Pull Requests](https://www.vidyasource.com/blog/beyond-github-culture-reconsider-pull-requests.md): Pull requests make sense in open-source but might be slowing your internal team down. Explore some more efficient alternatives. (HTML: https://www.vidyasource.com/blog/beyond-github-culture-reconsider-pull-requests/) - [It's Not About Unit Or Integration Tests. It's About Confidence](https://www.vidyasource.com/blog/not-about-unit-integration-tests-about-confidence.md): Forget traditional testing categories and focus on what builds confidence. Use AI to generate test data efficiently. (HTML: https://www.vidyasource.com/blog/not-about-unit-integration-tests-about-confidence/) - [How Sports Busts Common Myths About Software Engineering Teams](https://www.vidyasource.com/blog/how-sports-busts-common-myth-software-engineering-teams.md): Learn how sports history reveals why great engineering teams need more than star talent - they need systems, culture, and leadership that elevate everyone's performance. (HTML: https://www.vidyasource.com/blog/how-sports-busts-common-myth-software-engineering-teams/) - [No One Likes Cognitive Load: But Whose Cognitive Load?](https://www.vidyasource.com/blog/no-one-likes-cognitive-load-but-whose-cognitive-load.md): Minimizing cognitive load is more important than maximizing clean code, but cognitive load is subjective. Learn how to establish team standards that enhance productivity and maintainability. (HTML: https://www.vidyasource.com/blog/no-one-likes-cognitive-load-but-whose-cognitive-load/) - [Programming Without Ifs: Better Alternatives to Conditional Logic](https://www.vidyasource.com/blog/programming-without-ifs-better-alternatives-to-conditional-logic.md): Discover how to write more maintainable code by replacing conditional logic with practical alternatives that reduce complexity and improve your software." (HTML: https://www.vidyasource.com/blog/programming-without-ifs-better-alternatives-to-conditional-logic/) - [Waterfall's "Scope Creep" is Agile's Revelation](https://www.vidyasource.com/blog/waterfall-scope-creep-is-agile-revelation.md): Discover why "scope creep" isn't a problem but a valuable insight. Learn how agile methodology embraces change to build better products while Waterfall's rigid planning leads to missed opportunities. (HTML: https://www.vidyasource.com/blog/waterfall-scope-creep-is-agile-revelation/) - [Fibonacci Theater: Adding Mathematical Mystique to Scrum Estimation](https://www.vidyasource.com/blog/fibonacci-theater-adding-mathematical-mystique-to-scrum-estimation.md): Story points are the fundamental metric in Scrum, but using the Fibonacci Sequence to estimate them is more gimmick than substance. (HTML: https://www.vidyasource.com/blog/fibonacci-theater-adding-mathematical-mystique-to-scrum-estimation/) - [Vidya Talks Modernizing Tech Hiring in the Federal Government at the White House](https://www.vidyasource.com/blog/vidya-talks-modernizing-tech-hiring-federal-government-white-house.md): Vidya spoke to tech recruiters about how the federal government can hire and retain the best tech talent. (HTML: https://www.vidyasource.com/blog/vidya-talks-modernizing-tech-hiring-federal-government-white-house/) - [Lessons from The Bear for Tech Leaders](https://www.vidyasource.com/blog/lessons-from-the-bear-for-tech-leaders.md): Discover unexpected leadership insights from 'The Bear.' Learn how tech leaders can improve candidate experiences and foster employee success. Valuable lessons from an Emmy-winning series. (HTML: https://www.vidyasource.com/blog/lessons-from-the-bear-for-tech-leaders/) - [Modernizing Online Passport Renewal: Vidya's Success at the State Department](https://www.vidyasource.com/blog/modernizing-online-passport-renewal-vidya-success-at-the-state-department.md): Vidya helped develop the architecture for Online Passport Renewal, which is now live in beta at the State Department. (HTML: https://www.vidyasource.com/blog/modernizing-online-passport-renewal-vidya-success-at-the-state-department/) - [Vidya Awarded a GSA MAS Contract](https://www.vidyasource.com/blog/vidya-awarded-gsa-mas-contract.md): Vidya is proud that GSA has awarded us a Multiple Award Schedule contract for providing IT Professional Services to the government. (HTML: https://www.vidyasource.com/blog/vidya-awarded-gsa-mas-contract/) - [Vidya is on the Pod](https://www.vidyasource.com/blog/vidya-on-the-pod.md): Vidya President Neil Chaudhuri appeared on the Guidance Counselor 2.0 Podcast with Taylor Desseyn. (HTML: https://www.vidyasource.com/blog/vidya-on-the-pod/) - [The McKinsey Developer Productivity Analysis Is a Ruse](https://www.vidyasource.com/blog/the-mckinsey-developer-productivity-analysis-is-a-ruse.md): The McKinsey developer productivity analysis is worse than wrong. It's bad faith. Your organization can do so much better. (HTML: https://www.vidyasource.com/blog/the-mckinsey-developer-productivity-analysis-is-a-ruse/) - [Why Types and Tests are Both Essential in Programming](https://www.vidyasource.com/blog/why-types-and-tests-are-both-essential-in-programming.md): The arguments for types vs. tests is silly. You need both because they solve two different problems. (HTML: https://www.vidyasource.com/blog/why-types-and-tests-are-both-essential-in-programming/) - [Why Coding Interviews are the Worst](https://www.vidyasource.com/blog/why-coding-interviews-are-the-worst.md): Coding interviews are the worst not only for candidates but also for your business. Here's what you should do instead. (HTML: https://www.vidyasource.com/blog/why-coding-interviews-are-the-worst/) - [SWaM Certification: Vidya Is Certified in Virginia](https://www.vidyasource.com/blog/vidya-is-swam-certified-by-virginia-department-small-business-supplier-diversity.md): Vidya earned SWaM certification from the Virginia Department of Small Business and Supplier Diversity as a Small, Micro, Minority Owned, and 8(a) business. (HTML: https://www.vidyasource.com/blog/vidya-is-swam-certified-by-virginia-department-small-business-supplier-diversity/) - [Vidya is 8(a) Certified!](https://www.vidyasource.com/blog/vidya-is-8a-certified-by-us-small-business-administration.md): We are proud to be certified 8(a) by the US Small Business Administration. (HTML: https://www.vidyasource.com/blog/vidya-is-8a-certified-by-us-small-business-administration/) - [Which Programming Languages Should I Learn First?](https://www.vidyasource.com/blog/which-programming-languages-should-i-learn-first.md): When you're just starting out in tech, don't focus on programming languages but on something else entirely. (HTML: https://www.vidyasource.com/blog/which-programming-languages-should-i-learn-first/) - [What If...Marvel Built a Minimum Viable Product?](https://www.vidyasource.com/blog/what-if-marvel-built-a-minimum-viable-product.md): Product development should begin with an MVP, and Marvel's What if?... is exactly that. (HTML: https://www.vidyasource.com/blog/what-if-marvel-built-a-minimum-viable-product/) - [Lessons Learned from Building a React Component Library with TypeScript](https://www.vidyasource.com/blog/lessons-learned-react-component-library-typescript.md): Lessons learned, and not only about tech, from building a React component library for government. (HTML: https://www.vidyasource.com/blog/lessons-learned-react-component-library-typescript/) - [Scala vs Go: Who Wore It Better?](https://www.vidyasource.com/blog/scala-go.md): Scala vs Go compared on error handling, collections, and absent values. Which one fits your project? (HTML: https://www.vidyasource.com/blog/scala-go/) - [Code Coverage Is Killing You](https://www.vidyasource.com/blog/code-coverage-is-killing-you.md): Code coverage is intuitive but dangerous. There are quality metrics that are so much better. (HTML: https://www.vidyasource.com/blog/code-coverage-is-killing-you/) - [Dark Mode in Next.js using Tailwind CSS and React Hooks](https://www.vidyasource.com/blog/dark-mode-nextjs-tailwindcss-react-hooks.md): Use the power of Tailwind CSS and React Hooks to build Dark Mode users can control into your Next.js site. (HTML: https://www.vidyasource.com/blog/dark-mode-nextjs-tailwindcss-react-hooks/) - [Vidya 33 1/3](https://www.vidyasource.com/blog/vidya-thirty-three-third.md): The third iteration of the Vidya website is a Brave Verified Creator PWA with Dark Mode. Let's talk about it. (HTML: https://www.vidyasource.com/blog/vidya-thirty-three-third/) - [Highlights from CRMSDC Leaders and Legends](https://www.vidyasource.com/blog/highlights-from-crmsdc-leaders-and-legends-top-100-mbe-awards.md): CRMSDC Leaders and Legends was a blast, and it was an honor to receive a Top 100 MBE Award. (HTML: https://www.vidyasource.com/blog/highlights-from-crmsdc-leaders-and-legends-top-100-mbe-awards/) - [Welcoming CRMSDC](https://www.vidyasource.com/blog/welcoming-crmsdc-diversity-technology-consulting-minority-owned-business.md): We are proud to be certified by CRMSDC as a Minority Business Enterprise. (HTML: https://www.vidyasource.com/blog/welcoming-crmsdc-diversity-technology-consulting-minority-owned-business/) - [25K and Counting on Stack Overflow](https://www.vidyasource.com/blog/stack-overflow-java-scala-spark-diversity-25k-and-counting.md): It's an honor and a privilege to have helped 2.6M developers and earned 25K points on Stack Overflow. (HTML: https://www.vidyasource.com/blog/stack-overflow-java-scala-spark-diversity-25k-and-counting/) - [Welcoming USPAACC](https://www.vidyasource.com/blog/welcoming-uspaacc-diversity-technology-consulting-minority-owned-business.md): We are proud to be certified by USPAACC as an Asian and minority-owned small business. (HTML: https://www.vidyasource.com/blog/welcoming-uspaacc-diversity-technology-consulting-minority-owned-business/) - [How to Buy Cyber—Getting Started](https://www.vidyasource.com/blog/how-to-buy-cyber-getting-started.md): We need to empower procurement officials to take initiative when buying cybersecurity solutions. This is how. (HTML: https://www.vidyasource.com/blog/how-to-buy-cyber-getting-started/) - [Lessons from Java for Testing in React](https://www.vidyasource.com/blog/lessons-java-testing-react.md): See how years of testing in Java have taught us lessons you can apply to improve your testing in React. (HTML: https://www.vidyasource.com/blog/lessons-java-testing-react/) - [The Business Case for Functional Programming](https://www.vidyasource.com/blog/business-case-for-functional-programming.md): Learn how functional programming can make your teams more productive than you ever imagined. (HTML: https://www.vidyasource.com/blog/business-case-for-functional-programming/) - [The Signal and the Noise](https://www.vidyasource.com/blog/the-signal-and-the-noise.md): One senator made a great effort to hold social media accountable for 2016. We need more. (HTML: https://www.vidyasource.com/blog/the-signal-and-the-noise/) - [Talking Trends at Tech Talk DC](https://www.vidyasource.com/blog/talking-trends-at-tech-talk-dc.md): Hope to see you at our talk 'Here's What's Trending in Software Engineering.' (HTML: https://www.vidyasource.com/blog/talking-trends-at-tech-talk-dc/) - [It's Not Only About the Benjamins](https://www.vidyasource.com/blog/its-not-only-about-the-benjamins.md): The highest paying languages are also among the most fun and productive. We know from experience. (HTML: https://www.vidyasource.com/blog/its-not-only-about-the-benjamins/) - [Vidya Reloaded](https://www.vidyasource.com/blog/vidya-reloaded.md): Our new website is a Progressive Web Application. Here's why that's cool. (HTML: https://www.vidyasource.com/blog/vidya-reloaded/) - [Speaking at Code Writers Workshop 2017](https://www.vidyasource.com/blog/speaking-at-code-writers-workshop-2017.md): Hope to see you at our talk 'Here's What's Trending in Software Engineering.' (HTML: https://www.vidyasource.com/blog/speaking-at-code-writers-workshop-2017/) - [ScrumMaster++](https://www.vidyasource.com/blog/scrummaster-plus-plus-agile.md): A ScrumMaster with technical skills can use them without compromising the Scrum process. (HTML: https://www.vidyasource.com/blog/scrummaster-plus-plus-agile/) - [A New Strategy for Scala](https://www.vidyasource.com/blog/a-new-strategy-for-scala.md): The report of the death of the Strategy Pattern has been greatly exaggerated. (HTML: https://www.vidyasource.com/blog/a-new-strategy-for-scala/) - [Going Retro With Style](https://www.vidyasource.com/blog/going-retro-with-style.md): It isn't just engineers who have a lot to learn from agile retrospectives. (HTML: https://www.vidyasource.com/blog/going-retro-with-style/) - [Welcoming Webster & Fredrickson, PLLC](https://www.vidyasource.com/blog/welcoming-webster-and-fredrickson-pllc.md): We are lucky to work with Webster & Fredrickson, PLLC, a premier DC law firm. (HTML: https://www.vidyasource.com/blog/welcoming-webster-and-fredrickson-pllc/) - [Clowns to the Left of Me, Jokers to the Right](https://www.vidyasource.com/blog/clowns-to-the-left-of-me-jokers-to-the-right.md): The caricature of the TSA Randomizer was ripe for mockery, but the reality is complicated. (HTML: https://www.vidyasource.com/blog/clowns-to-the-left-of-me-jokers-to-the-right/) - [Mo Incentives Mo Problems](https://www.vidyasource.com/blog/mo-incentives-mo-problems.md): As government IT becomes more agile, it's time to rethink incentives in contracts. (HTML: https://www.vidyasource.com/blog/mo-incentives-mo-problems/) - [ All We Do Is Win Win Win](https://www.vidyasource.com/blog/all-we-do-is-win-win-win.md): The President issued a challenge to make technology work for government. We won it. (HTML: https://www.vidyasource.com/blog/all-we-do-is-win-win-win/) - [Analytics With Apache Spark Is Here](https://www.vidyasource.com/blog/analytics-with-apache-spark-is-here.md): Learn the hottest technology in data analytics with our Apache Spark course. (HTML: https://www.vidyasource.com/blog/analytics-with-apache-spark-is-here/) - [Talking Scala](https://www.vidyasource.com/blog/talking-scala.md): I will be speaking on Scala at Polyglot Programming DC! (HTML: https://www.vidyasource.com/blog/talking-scala/) - [This Week In #Scala](https://www.vidyasource.com/blog/this-week-in-scala.md): Our latest tutorial made it to This Week in #Scala! (HTML: https://www.vidyasource.com/blog/this-week-in-scala/) - [When You Do Want None](https://www.vidyasource.com/blog/when-you-do-want-none.md): A quick tip for Scala developers for handling empty and whitespace strings in their code. (HTML: https://www.vidyasource.com/blog/when-you-do-want-none/) - [Zero Tolerance: Checking a Java BigDecimal for Zero](https://www.vidyasource.com/blog/zero-tolerance-java.md): Is your Java BigDecimal zero? Compare signum, compareTo, and equals to find the check that works. (HTML: https://www.vidyasource.com/blog/zero-tolerance-java/) - [No Experience? No Problem!](https://www.vidyasource.com/blog/no-experience-no-problem.md): You really don't need professional experience with a technology to get a job in it. Just passion. (HTML: https://www.vidyasource.com/blog/no-experience-no-problem/) - [Canstruction DC 2014](https://www.vidyasource.com/blog/canstruction-dc-2014.md): We are thrilled to have partnered with Dewberry to fight hunger at Canstruction DC 2014. (HTML: https://www.vidyasource.com/blog/canstruction-dc-2014/) - [Know Your Options](https://www.vidyasource.com/blog/know-your-options-scala-java.md): Null is a real pain, but functional languages like Scala can make it a lot easier. (HTML: https://www.vidyasource.com/blog/know-your-options-scala-java/) - [Welcoming Madena Solutions](https://www.vidyasource.com/blog/welcoming-madena-solutions.md): We are lucky to be working with Madena, a premier provider of consulting services in the Medicare space. (HTML: https://www.vidyasource.com/blog/welcoming-madena-solutions/) - [You Must Unlearn What You Have Learned](https://www.vidyasource.com/blog/you-must-unlearn-what-you-have-learned.md): Embrace the wisdom of Yoda to succeed in technology and project management. (HTML: https://www.vidyasource.com/blog/you-must-unlearn-what-you-have-learned/) - [Lighting a Spark with HBase](https://www.vidyasource.com/blog/lighting-spark-hbase.md): Apache Spark is great for Hadoop analytics, and it works just fine with HBase. (HTML: https://www.vidyasource.com/blog/lighting-spark-hbase/) - [Idiom Savant](https://www.vidyasource.com/blog/idiom-savant-java-ruby.md): Idioms are just as important in programming languages as they are in spoken languages. (HTML: https://www.vidyasource.com/blog/idiom-savant-java-ruby/) - [Don't Go Chasing Waterfall](https://www.vidyasource.com/blog/dont-go-chasing-waterfall.md): Michael Daconta happily blames agile software development for the problems on the health care website. He's wrong. And yes, the title is a TLC reference. (HTML: https://www.vidyasource.com/blog/dont-go-chasing-waterfall/) - [Java is Dysfunctional with Big Data](https://www.vidyasource.com/blog/java-is-dysfunctional-with-big-data.md): Java is great, but there are far better options for big data analytics. (HTML: https://www.vidyasource.com/blog/java-is-dysfunctional-with-big-data/) - [First Contact](https://www.vidyasource.com/blog/first-contact.md): Welcome to Vidya (HTML: https://www.vidyasource.com/blog/first-contact/) --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "About Vidya" description: "Who Vidya is, what it values, and who leads it." canonical: https://www.vidyasource.com/about/ type: page --- # About Vidya > Who Vidya is, what it values, and who leads it. - Canonical page: https://www.vidyasource.com/about/ - Structured data (JSON-LD): https://www.vidyasource.com/about.json ## About Vidya Vidya is a software services and consulting company near Washington, DC, USA. ### Our Mission Vidya builds modernized on-premise, cloud, and hybrid solutions for commercial businesses and government agencies according to their strategic objectives. Vidya's work proves there is no need to choose between continuous delivery and high quality. Federal SBA 8(a), Commonwealth of Virginia SWaM, and multiple commercial certifications as a minority-owned business are testament to Vidya's ability to deliver lean, innovative solutions for clients and to do so the right way using techniques like agile development, DevSecOps, and automated testing for functionality, security, accessibility, and performance. ### Our Differentiators - Proven track record of modernizing cloud-scale software by integrating leading-edge technologies with legacy systems. For example, Vidya helped the State Department reduce passport renewal time from months to days with 97% customer satisfaction and 80% increased trust in government. - Dedication to communication as the basis for teamwork, transparency, accountability, and humility. - Demonstrated ability to translate complex technical concepts for non-technical stakeholders. - Small business agility combined with enterprise-level delivery experience. ### Our Values: Great Technology Needs All of Us Vidya is a passionate advocate for diversity in technology. Prejudice against anyone on the basis of gender, race, ethnicity, nationality, religion, orientation, ability, or even academic background not only limits individual growth but also limits the creative energy necessary to develop the best software. President Neil Chaudhuri leads this commitment publicly. ### Our Name Vidya comes from the Sanskrit for "right knowledge" or "clarity." The name embodies the learning spirit that is the essence of software engineering. Vidya is associated with the Hindu deity Saraswati, the Goddess of Learning, often depicted wearing a spotless white sari and riding a white swan. The swan is Vidya's mark. ### Leadership Neil Chaudhuri is President of Vidya, a recognized speaker on software engineering trends, and an active contributor to the engineering community with significant Stack Overflow reputation. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Contact Vidya" description: "How to reach Vidya about consulting, training, or speaking." canonical: https://www.vidyasource.com/contact/ type: page --- # Contact Vidya > How to reach Vidya about consulting, training, or speaking. - Canonical page: https://www.vidyasource.com/contact/ - Structured data (JSON-LD): https://www.vidyasource.com/contact.json - Email: info@vidyasource.com - Phone: +1-202-430-5695 - Mail: 3057 Nutley Street #285, Fairfax, VA 22031, USA - Web form: https://www.vidyasource.com/contact/ (protected by Cloudflare Turnstile, so it needs a browser) Vidya answers every inquiry personally. Say what you are trying to modernize, build, or learn, and Vidya will reply with next steps. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "AI Consulting & Agentic Engineering" description: "Vidya builds AI systems that succeed because they're grounded in your actual data and processes, engineered for safety, reliability, and cost control." canonical: https://www.vidyasource.com/consulting/ai-consulting/ type: service tags: ["AI", "Agentic AI", "Agents", "MCP", "LLMs", "Agent Skills", "Machine Learning"] image: https://www.vidyasource.com/img/courses/enterprise-ai.webp --- # AI Consulting & Agentic Engineering > Vidya builds AI systems that succeed because they're grounded in your actual data and processes, engineered for safety, reliability, and cost control. - Canonical page: https://www.vidyasource.com/consulting/ai-consulting/ - Structured data (JSON-LD): https://www.vidyasource.com/consulting/ai-consulting.json - Topics: AI, Agentic AI, Agents, MCP, LLMs, Agent Skills, Machine Learning ## Frequently asked questions **Why does Vidya emphasize business context over computing power?** Because that's where most AI projects actually fail. MIT found that 95% of enterprise AI deployments fail for business reasons, like missing data strategy and unclear goals. Gartner separately reports that organizations who prioritize context can lift AI accuracy by up to 80% and cut costs by up to 60%. Vidya applies a simple test before recommending any AI project. If the information an agent needs can't realistically be made available to it, the task isn't ready for AI yet. **What is the Agentic AI Foundation, and why does it matter to my business?** It's the Linux Foundation-backed group that now governs the open standards behind today's AI tools, similar to the role the W3C played for the early web. Vidya's founder is one of its inaugural 138 Ambassadors. Building your AI strategy on those open standards means upgrading to a better tool later is a simple configuration change. **How does Vidya approach AI safety and bias?** Vidya's position is that an autonomous agent belongs in production only when a human can plan, trace, and reverse every action it takes, and every high-impact system Vidya builds keeps a fallback path that works without the AI. On bias, Vidya grounds its practice in the research of Dr. Timnit Gebru and Dr. Joy Buolamwini, favoring smaller, better-documented models and testing outputs before they reach a decision that affects a real person. **How does Vidya keep AI reliable enough for tasks that must be correct every time?** By pairing the AI agent's language reasoning with a deterministic, rule-based step for the parts of a job that must produce the same correct answer every time, like a calculation or a compliance check, and tracking accuracy against real outcomes on an ongoing basis so drift gets caught early. **Does adding AI increase our operating costs?** Not automatically. Gartner reports that organizations who prioritize context can cut AI costs by up to 60%, and Vidya's deterministic, rule-based components cost a fraction of a full model call and never drift in price, so the parts of a system that don't need a language model don't pay for one. **How fast can Vidya show results?** Vidya scopes a first engagement as a two-week discovery sprint that ends in a working system connected to your own data. Vidya is a modernization firm that has spent three years helping clients turn AI from a pilot project into a system their teams actually rely on. Clients who bring Vidya an AI project get something working inside their own operations within weeks, built around how the business already runs. Most AI projects fail for reasons that have nothing to do with which model a company picks. MIT reported that 95% of enterprise AI deployments fail, largely from missing data strategy and unclear business goals. Vidya made the same case in [Python Is Not the Language of AI](https://www.vidyasource.com/blog/python-is-not-language-of-ai/). Succeeding with AI is a matter of giving a system a [culture of context](https://www.vidyasource.com/blog/create-a-culture-of-context-for-ai/). Gartner agrees, finding that prioritizing context can lift AI accuracy by up to 80% and cut costs by up to 60%. A simple test guides every AI recommendation Vidya makes. If the information an agent needs isn't realistically available to it, the task isn't ready for AI yet. Vidya builds AI safety and reliability directly into how a system runs. An autonomous agent belongs in production only when a human can plan, trace, and reverse every action it takes, with an oversight point built into each step. Every high-impact system keeps a fallback path that works without the AI. Where part of a job must produce the same correct answer every time, like a calculation or a compliance check, Vidya pairs the AI agent's language reasoning with a deterministic, rule-based step built for that part. Vidya also tracks accuracy against real outcomes on an ongoing basis, so drift gets caught early. On bias, Vidya grounds its practice in the research of Dr. Timnit Gebru and Dr. Joy Buolamwini, favoring smaller, better-documented models and testing outputs before they reach a decision that affects a real person. The Agentic AI Foundation, the Linux Foundation-backed group governing the open standards behind today's leading AI tools, named Vidya's founder to its inaugural cohort of 138 Ambassadors across 41 countries. A stack built around one vendor's product fails the day that vendor deprecates it or ships a worse model. Vidya builds on open, vendor-neutral standards, so upgrading later becomes a simple configuration change that protects a client's investment. The same deterministic components that keep a system reliable also keep it cheap to run. A rule-based step costs a fraction of a model call and never drifts in price. Vidya scopes a first AI engagement as a two-week discovery sprint that identifies one high-value business process worth automating and maps the information it needs. At the end of those two weeks, you get a working system connected to your own data. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Legacy System Modernization" description: "Vidya modernizes legacy code, data architecture, and delivery culture in small safe steps, the approach behind Online Passport Renewal and Recreation.gov." canonical: https://www.vidyasource.com/consulting/legacy-system-modernization/ type: service tags: ["Architecture", "Microservices", "Technical Debt", "Continuous Delivery", "Data", "DevSecOps"] image: https://www.vidyasource.com/img/blog/passport.jpg --- # Legacy System Modernization > Vidya modernizes legacy code, data architecture, and delivery culture in small safe steps, the approach behind Online Passport Renewal and Recreation.gov. - Canonical page: https://www.vidyasource.com/consulting/legacy-system-modernization/ - Structured data (JSON-LD): https://www.vidyasource.com/consulting/legacy-system-modernization.json - Topics: Architecture, Microservices, Technical Debt, Continuous Delivery, Data, DevSecOps ## Frequently asked questions **What is Vidya's flagship legacy system modernization result?** Online Passport Renewal for the U.S. Department of State, where Vidya's work cut passport renewal from months to days, produced a 97% customer satisfaction rate, and lifted public trust in the service by 80%. The service has processed more than 2 million online renewals. **Does legacy system modernization mean only rewriting code?** No. The data model underneath a legacy application usually encodes the same outdated assumptions the code does, so Vidya treats data architecture as part of the same engagement. The delivery culture counts just as much, because a team that ships once a quarter through a manual release cannot safely operate a modern architecture. Vidya builds the agile practice and the DevSecOps pipeline alongside the new code. **How does Vidya modernize a system without disrupting current users?** Vidya peels one function out of the legacy system at a time and runs the old and new versions side by side against the same live traffic, retiring the old path only once the new one has proven itself, so customers never see a service interruption. **How long does a typical modernization engagement take?** It depends on scope. Vidya's Recreation.gov modernization took 6 months and tripled national park reservation volume, while the Consular Systems Modernization program has run for 6 years and continues to expand. Legacy systems slow down more than IT. They slow down every process built on top of them, from customer approvals to safety reviews. Vidya has spent 6 years fixing that on systems millions of people depend on. Organizations that hire Vidya keep serving their current users while Vidya rebuilds the parts that are failing them. Rewriting code alone rarely fixes a legacy program. The data model underneath usually encodes the same assumptions the application does, so Vidya treats [data architecture](https://www.vidyasource.com/consulting/software-architecture/) as part of the same job and decouples readers from writers before the services multiply. Delivery culture carries just as much weight. A team that ships once a quarter through a manual release cannot safely operate a modern architecture, so Vidya builds the agile practice and the [DevSecOps pipeline](https://www.vidyasource.com/consulting/cloud-native-devsecops/) alongside the code. The clearest proof point is [Online Passport Renewal](https://www.vidyasource.com/case-studies/consular-systems-modernization-state-department/) for the U.S. Department of State's Bureau of Consular Affairs, part of a program Vidya has supported since 2019. That work cut passport renewal from months to days. The most recent measurement puts customer satisfaction at 97% and the increase in public trust at 80%, with more than 2 million renewals processed online. Vidya delivered a similar result at Recreation.gov, where a 6-month modernization tripled national park reservation volume. At [HealthCare.gov](https://www.vidyasource.com/case-studies/healthcare-gov-modernization/), a technology overhaul improved performance by more than 30% while Vidya mentored more than 20 developers through the transition. On these programs Vidya peels one function out of the legacy system at a time and runs the old and new versions side by side against the same live traffic. Vidya retires the old path only once the new one has proven itself. Vidya scopes an engagement with a 4-week assessment that names your system's biggest risks and sequences the fixes safely. You get a written roadmap and visible progress in production within the first quarter. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Cloud Native Architecture & DevSecOps" description: "Vidya builds cloud systems on AWS and Azure with automated, secure release pipelines that shipped the State Department's passport service and tripled Recreation.gov's reservation volume." canonical: https://www.vidyasource.com/consulting/cloud-native-devsecops/ type: service tags: ["DevSecOps", "DevOps", "Kubernetes", "Docker", "AWS", "SRE", "Microservices"] image: https://www.vidyasource.com/img/blog/recreation.png --- # Cloud Native Architecture & DevSecOps > Vidya builds cloud systems on AWS and Azure with automated, secure release pipelines that shipped the State Department's passport service and tripled Recreation.gov's reservation volume. - Canonical page: https://www.vidyasource.com/consulting/cloud-native-devsecops/ - Structured data (JSON-LD): https://www.vidyasource.com/consulting/cloud-native-devsecops.json - Topics: DevSecOps, DevOps, Kubernetes, Docker, AWS, SRE, Microservices ## Frequently asked questions **Which clouds does Vidya work in?** Both AWS and Azure. Vidya has run production systems for the Department of State, a national park reservation system, and a threat-detection system across both, so a client isn't locked into a single provider. **How does Vidya decide on infrastructure for a project?** Vidya matches the setup to a client's actual traffic and team size. A small team gets a simple setup, and a large program with dozens of services gets the more robust platform that scale actually requires. **What is a Golden Path, and why does Vidya build one?** A Golden Path is a self-service release platform where a team's process comes with an organization's security and compliance rules already built in, so no team has to assemble them on its own. That discipline is why, when the Log4Shell vulnerability broke in 2021, Vidya could find every place the affected library appeared across its codebases and patch it within hours. Vidya has built and operated cloud systems for clients since 2010, and the teams that hire Vidya for this work get software releases that happen on a predictable schedule. Your team ships updates as a routine, low-stress part of the week. The Consular Systems Modernization program at the Department of State now runs on an automated release pipeline Vidya built with security checks in from the start. That pipeline supports a system that serves the public every day. Recreation.gov saw a similar result. A 6-month modernization supported a threefold increase in reservation volume without a matching increase in outages. Vidya builds this kind of pipeline as a self-service platform, a Golden Path, so a team's release process comes with an organization's security and compliance rules already built in. That discipline paid off when the Log4Shell vulnerability broke in 2021. Because Vidya already tracked a software bill of materials for its systems, Vidya found every place the affected library appeared across its codebases and patched it within hours. Vidya scopes this kind of engagement with a 2-week audit that measures how often your team currently ships updates and how quickly it recovers from a problem. You get a prioritized list of fixes and one automated release process live within the first 6 weeks. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Software & Data Architecture" description: "Vidya designs systems built to be understood and extended by the client's own team, the same discipline behind a shared toolkit used across 6 State Department product teams." canonical: https://www.vidyasource.com/consulting/software-architecture/ type: service tags: ["Architecture", "Microservices", "Security"] image: https://www.vidyasource.com/img/consulting/development.jpg --- # Software & Data Architecture > Vidya designs systems built to be understood and extended by the client's own team, the same discipline behind a shared toolkit used across 6 State Department product teams. - Canonical page: https://www.vidyasource.com/consulting/software-architecture/ - Structured data (JSON-LD): https://www.vidyasource.com/consulting/software-architecture.json - Topics: Architecture, Microservices, Security ## Frequently asked questions **How does Vidya decide how complex a system should be?** Vidya starts with the simplest design that solves the problem in front of it and adds complexity only when a real business need calls for it. **What does Vidya mean by treating data as something each team owns?** Each business team owns and is responsible for keeping its own data accurate and well documented. That gives both people and AI tools a clear, trustworthy way to find what a piece of data actually means. **What has Vidya's architecture work delivered at scale?** On the Consular Systems Modernization program, Vidya found that passport renewal, reporting a birth abroad, and visa processing all needed the same identity verification, case management, and document generation underneath. Vidya built each of those once and let 6 separate product teams reuse them behind one API layer, cutting the time to build a new screen by 40% while meeting federal accessibility requirements. Vidya has architected large-scale systems for over a decade, and the organizations that hire Vidya for architecture consulting get a system their own team can maintain and grow long after Vidya leaves. Your team inherits decisions it can actually understand, because Vidya documents why the system is built the way it is. The Consular Systems Modernization program at the Department of State now runs on a shared set of reusable building blocks Vidya built for 6 separate product teams. That work cut the time it took each team to build a new screen by 40% while meeting federal accessibility requirements. Vidya found that shared foundation by looking past the surface differences between passport renewal, reporting a birth abroad, and visa processing. Underneath, all three needed the same identity verification, case management, and document generation, so Vidya built each of those once and let every team reuse them behind one API layer. Vidya starts every project with the simplest design that solves the problem in front of it. Vidya adds complexity, like splitting a system into smaller independent pieces, only at the point where a real business need calls for it. For data specifically, [Vidya's approach](https://www.vidyasource.com/blog/create-a-culture-of-context-for-ai/) treats each business team's data as a product that team owns and keeps accurate and well documented. Vidya scopes an architecture engagement as a 3-week assessment that identifies where your current system slows your team down and produces a plan your own engineers agree makes sense. You get that plan and a sequenced path to get there. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Technology Culture & Organizational Consulting" description: "Vidya builds engineering culture and delivery practices backed by real metrics, agile credentials, and a real modernization track record at HealthCare.gov." canonical: https://www.vidyasource.com/consulting/technology-culture/ type: service tags: ["Agile", "Scrum", "Project Management", "PMP"] image: https://www.vidyasource.com/img/blog/tech-hiring.png --- # Technology Culture & Organizational Consulting > Vidya builds engineering culture and delivery practices backed by real metrics, agile credentials, and a real modernization track record at HealthCare.gov. - Canonical page: https://www.vidyasource.com/consulting/technology-culture/ - Structured data (JSON-LD): https://www.vidyasource.com/consulting/technology-culture.json - Topics: Agile, Scrum, Project Management, PMP ## Frequently asked questions **What sets Vidya's approach to team culture apart?** Vidya's founder holds Certified ScrumMaster and Certified Scrum Professional credentials, and built the Modern Agile training course around the same leadership, trust, and psychological safety principles Vidya applies in every culture engagement. **How does Vidya measure a team's actual delivery quality?** With DORA and SPACE, two widely used research frameworks for delivery speed and team well-being, because a certification describes a process on paper while these metrics describe the work itself. **What proof does Vidya have that this approach changes outcomes?** At HealthCare.gov, Vidya mentored more than 20 developers through a major technology transition as part of a modernization that improved performance by more than 30%, gains the client's own team kept after Vidya left. **Does Vidya offer this as a training course?** Yes. Vidya's Modern Agile course moves teams beyond checklist-style process toward the practices that actually change how fast a team ships, taught by the same team that runs Vidya's consulting engagements. Vidya has learned that a technology transformation succeeds only when the people around it trust each other enough to raise a problem early, before it becomes an outage. The clients who hire Vidya for culture work get a team that ships faster because bad news travels quickly and cheaply, while it is still easy to fix. Two widely used research frameworks, DORA and SPACE, anchor how Vidya measures a team's actual delivery speed and well-being, because a process certification only describes what a team promised to do. Vidya's founder also holds Certified ScrumMaster and Certified Scrum Professional credentials. At HealthCare.gov, Vidya mentored more than 20 developers through a major technology transition as part of a modernization that improved performance by more than 30%. That is proof that a team keeps the gains after the consultant leaves, because the people who own the work understand why it changed. Vidya built its Modern Agile training course around that same idea. A team ships faster once its leadership builds the trust and psychological safety that let people surface bad news early. Vidya scopes a culture engagement as a 3-week assessment covering your team structure, your recruiting pipeline, and how fast your team actually ships work today. You leave with a prioritized set of changes and, if your team wants it, a seat in the next Modern Agile cohort. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "AI & Modernization Consulting for Government" description: "Vidya is an 8(a) and SWaM-certified firm on GSA MAS schedule 47QTCA24D0069 (SIN 54151S), delivering modernization and AI work for the State Department and other federal agencies." canonical: https://www.vidyasource.com/consulting/government-modernization/ type: service tags: ["Government", "Partners", "Project Management", "AI"] image: https://www.vidyasource.com/img/blog/gsa-landscape.jpg --- # AI & Modernization Consulting for Government > Vidya is an 8(a) and SWaM-certified firm on GSA MAS schedule 47QTCA24D0069 (SIN 54151S), delivering modernization and AI work for the State Department and other federal agencies. - Canonical page: https://www.vidyasource.com/consulting/government-modernization/ - Structured data (JSON-LD): https://www.vidyasource.com/consulting/government-modernization.json - Topics: Government, Partners, Project Management, AI ## Frequently asked questions **What federal contract vehicles can agencies use to bring on Vidya?** Vidya holds GSA Multiple Award Schedule (MAS) contract 47QTCA24D0069 under SIN 54151S for IT Professional Services, in addition to teaming and subcontract arrangements on existing programs. **What certifications does Vidya carry?** Vidya is SBA 8(a) certified, SWaM (Small, Women-owned, and Minority-owned) certified through the Commonwealth of Virginia, a National Minority Supplier Development Council (NMSDC)-certified minority business enterprise, and US Pan Asian American Chamber of Commerce (USPAACC) certified. **What is Vidya's largest delivered federal engagement?** The Consular Systems Modernization program for the Department of State's Bureau of Consular Affairs, a $434 million prime contract where Vidya's work delivered Online Passport Renewal, cutting renewal time from months to days with a 97% customer satisfaction rate. **Will I get Vidya's senior engineers, or a staffing pyramid?** The engineers who scope your program are the same engineers who do the work. There is no staffing pyramid and no handoff to a junior bench after award. Vidya is a certified 8(a) small business under the U.S. Small Business Administration (SBA) and SWaM (Small, Women-owned, and Minority-owned) certified through the Commonwealth of Virginia, and has delivered federal modernization work since 2019. The engineers who scope your program are the same engineers who do the work, with no staffing pyramid and no handoff to a junior bench after award. Your program office gets a partner who can operate inside GSA Multiple Award Schedule (MAS) contract 47QTCA24D0069 from day one. Consular Systems Modernization for the Department of State's Bureau of Consular Affairs is Vidya's flagship federal program, a $434 million prime contract. Vidya's work under that contract delivered [Online Passport Renewal](https://www.vidyasource.com/blog/modernizing-online-passport-renewal-vidya-success-at-the-state-department/) and an earlier service for citizens reporting a child's birth abroad. That work cut passport renewal from months to days, produced a 97% customer satisfaction rate and an 80% increase in public trust, and has processed more than 2 million online renewals. Recreation.gov saw a comparable result, tripling national park reservation volume in a 6-month engagement, and HealthCare.gov improved performance by more than 30% under a similar modernization. Every government system Vidya builds runs on cloud platforms the federal government has already authorized for use, so agencies get modern infrastructure without taking on new security risk. Vidya's founder also brings two credentials specific to government buyers: membership in the Government Accountability Office's (GAO) Agile Expert Group, and authorship of the training that became DITAP (Digital IT Acquisition Professional), still used to teach federal acquisition staff how to buy software well. Vidya scopes a new federal engagement with a 2-week call that maps your program's requirements to a proposed team under GSA MAS 47QTCA24D0069. You get a tailored technical approach ready for your acquisition process within 30 days. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Cutting U.S. Passport Renewal From Months to Days" description: "The Bureau of Consular Affairs cut passport renewal from months to days. A Consular Systems Modernization case study on Vidya's architecture work at State." canonical: https://www.vidyasource.com/case-studies/consular-systems-modernization-state-department/ type: caseStudy tags: ["Modernization", "Government", "Architecture", "Java", "Security", "Java", "Spring Boot", "Red Hat OpenShift", "Salesforce", "Oracle Exadata", "RabbitMQ", "Resilience4j", "React", "TypeScript", "Chakra UI", "Azure Databricks", "Jenkins"] image: https://www.vidyasource.com/img/blog/passport.jpg --- # Cutting U.S. Passport Renewal From Months to Days > The Bureau of Consular Affairs cut passport renewal from months to days. A Consular Systems Modernization case study on Vidya's architecture work at State. - Canonical page: https://www.vidyasource.com/case-studies/consular-systems-modernization-state-department/ - Structured data (JSON-LD): https://www.vidyasource.com/case-studies/consular-systems-modernization-state-department.json - Topics: Modernization, Government, Architecture, Java, Security, Java, Spring Boot, Red Hat OpenShift, Salesforce, Oracle Exadata, RabbitMQ, Resilience4j, React, TypeScript, Chakra UI, Azure Databricks, Jenkins - Client: U.S. Department of State, Bureau of Consular Affairs - Sector: Federal Government - Engagement: Multi-year engagement - Technologies: Java, Spring Boot, Red Hat OpenShift, Salesforce, Oracle Exadata, RabbitMQ, Resilience4j, React, TypeScript, Chakra UI, Azure Databricks, Jenkins ## Results - **2 million+ online renewals.** Americans completed more than 2 million passport renewals through the online service in its first months. - **Months to days.** Renewal that once moved paper between offices for months now returns a passport in days, transforming the experience for millions. - **97% positive review, 80% trust lift.** Bureau surveys found 97 percent of respondents rated Online Passport Renewal positively, and 80 percent said the experience increased their trust in government. - **40% less UI development time.** Vidya's accessible React component library served 6 State Department product teams and cut their user interface development time by 40 percent while holding Section 508 conformance. ## Frequently asked questions **What is the Consular Systems Modernization program?** Consular Systems Modernization (CSM) is the U.S. Department of State program that moved the Bureau of Consular Affairs off legacy on-premise systems and onto a hybrid FedRAMP cloud. The State Department awarded the prime contract to Northrop Grumman at $434,248,871 across the base period and all options, and Peraton assumed the prime role after acquiring the division that held the work. Vidya performed on CSM as a subcontractor. **What was Vidya's role in the online passport renewal modernization?** Vidya supplied lead architecture, senior software engineering, and DevSecOps engineering on the FedRAMP High program that produced Online Passport Renewal. Vidya's engineers designed the event-driven Java and Spring Boot services on Red Hat OpenShift, built the accessible React component library the product teams shared, and configured the delivery pipeline. The State Department led the program and owns the outcome, and Vidya delivered a defined portion of the engineering underneath it. **How does Vidya approach State Department legacy system modernization?** Vidya peels one functional domain at a time out of the legacy estate behind a network traffic interceptor, runs the old and new paths in parallel with synchronized data, and retires the legacy path only after the replacement proves itself in production. That sequence costs more than a single cutover because the program pays for two systems at once. A service that issues over 20 million passports a year cannot absorb a failed flag day, so the duplication buys real insurance. **What results did Online Passport Renewal deliver for citizens?** Americans renewed more than 2 million passports online in the service's first months. Bureau surveys recorded a 97 percent positive review rate and an 80 percent increase in trust in government among users. The Partnership for Public Service named the State Department leaders and the Online Passport Renewal team Service to America Medals honorees. ## The Challenge The Bureau of Consular Affairs carries one of the heaviest citizen-service loads in the federal government. Its officers issue passports, adjudicate visas, and record Consular Reports of Birth Abroad across more than 270 diplomatic posts. They also staff crisis management around the clock. Passport demand alone runs 20 to 24 million applications a year. The bureau ran that volume on a process it designed decades ago for roughly 3 million passports a year. The figure later passed 23 million, and applicants waited months while paper moved between offices. Siebel case management, on-premise SOAP services, and decades-old databases held the whole thing together. Assistant Secretary Rena Bitter later called the fix "the most significant innovation in passports in 50 years." Earlier modernization attempts had failed, and that history left institutional skepticism to overcome before any architecture decision mattered. The State Department awarded the Consular Systems Modernization (CSM) prime contract to Northrop Grumman at $434,248,871 across the base period and all options, and Peraton later assumed the prime role. Vidya joined as a subcontractor on this FedRAMP High engagement, providing lead architecture, senior software engineering, and DevSecOps engineering. ## Vidya's Approach A single cutover on a service that issues over 20 million passports a year risks a national outage, so Vidya peeled functional domains out of the legacy systems one at a time behind a network traffic interceptor. The old and new paths ran in parallel with synchronized data, and the program retired a legacy path only after its replacement proved out in production. Parallel operation is tricky for many reasons but primarily because it's important to maintain clear sources of truth and materialize, rather than copy, data wherever possible. Passport renewal for millions of Americans could not pause for a modernization window, so understanding distributed systems architectural practices was paramount. The resulting architecture centers on events. New cases originate in Salesforce, and Salesforce Platform Events trigger a pipeline of Java and Spring Boot microservices on Red Hat OpenShift, with one service orchestrating state in a Saga-like pattern. Case data lands in Red Hat object storage and an Oracle Exadata Authoritative Data Store, and Oracle GoldenGate feeds a predictive analytics platform on Azure Databricks. Vidya built internal messaging on RabbitMQ with a schema registry and applied Resilience4j retries and circuit breakers so one slow dependency never cascades into a public outage. Zero Trust under Executive Order 14028 covers the path with encryption at rest and mutual TLS in transit. Six State Department product teams shared the same front end, so the bureau needed one accessible user interface foundation rather than six divergent ones. Vidya delivered a [React component library with TypeScript](https://www.vidyasource.com/blog/lessons-learned-react-component-library-typescript/) on Chakra UI, themed to the program's design system and tested with role-based selectors, so a component that screen readers cannot reach fails its own test suite. Those teams cut UI development time by 40 percent while holding Section 508 compliance. ## What Changed Americans now renew a passport from a browser. The service launched to the public, and more than 2 million people renewed online in its first months. Bureau surveys recorded a 97 percent positive review rate and an 80 percent increase in trust in government among the people who used it. Renewal that once consumed months returns a document in days. Earlier in the same engagement, American parents overseas gained eCRBA, an online path to a birth record for a child born abroad. The Partnership for Public Service named the State Department leaders and the Online Passport Renewal team Service to America Medals honorees. A legacy paper process reached a hybrid FedRAMP cloud without a single day off, and that method transfers to any federal program sitting on aging case management. Vidya applied the same incremental sequence for CMS on [HealthCare.gov](https://www.vidyasource.com/case-studies/healthcare-gov-modernization/), and it anchors the [government modernization](https://www.vidyasource.com/consulting/government-modernization/) and [legacy system modernization](https://www.vidyasource.com/consulting/legacy-system-modernization/) practices today. Name the one citizen-facing service your program office hears about most, and Vidya will return a target architecture and a domain-by-domain migration sequence for it within one quarter. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Making HealthCare.gov More Than 30% Faster for CMS" description: "CMS saw HealthCare.gov performance rise more than 30% after Vidya's Java to Scala rebuild on AWS. A federal healthcare system modernization case study." canonical: https://www.vidyasource.com/case-studies/healthcare-gov-modernization/ type: caseStudy tags: ["Modernization", "Government", "Architecture", "Java", "Scala", "Functional Programming", "Testing", "Scala", "Java", "Play Framework", "Spring", "AWS", "Java Virtual Machine", "ScalaTest", "ScalaCheck"] image: https://www.vidyasource.com/img/blog/health-care-gov.jpg --- # Making HealthCare.gov More Than 30% Faster for CMS > CMS saw HealthCare.gov performance rise more than 30% after Vidya's Java to Scala rebuild on AWS. A federal healthcare system modernization case study. - Canonical page: https://www.vidyasource.com/case-studies/healthcare-gov-modernization/ - Structured data (JSON-LD): https://www.vidyasource.com/case-studies/healthcare-gov-modernization.json - Topics: Modernization, Government, Architecture, Java, Scala, Functional Programming, Testing, Scala, Java, Play Framework, Spring, AWS, Java Virtual Machine, ScalaTest, ScalaCheck - Client: Centers for Medicare & Medicaid Services (CMS) - Sector: Federal Government - Engagement: 3-year engagement - Technologies: Scala, Java, Play Framework, Spring, AWS, Java Virtual Machine, ScalaTest, ScalaCheck ## Results - **More than 30% performance improvement.** HealthCare.gov ran more than 30 percent faster after Vidya rebuilt the Spring application in Scala on Play Framework, letting CMS absorb the open enrollment surge on hardware it already owned. - **20+ Java developers mentored in Scala.** Engineers on the client team learned the new language and its safety habits from the one person on the project who already knew them, so CMS kept the skills in house instead of renting them back every time the rules changed. - **Incomplete applications handled by design.** Every eligibility formula had to state which inputs could be missing, so a partial record became a business decision made in minutes during development instead of a crash, a help desk call, and hours of engineer time reproducing it. ## Frequently asked questions **What was Vidya's role in the HealthCare.gov modernization?** Vidya worked as a subcontractor to Accenture Federal Services, which held the prime relationship with the Centers for Medicare & Medicaid Services and carried the overall program. Vidya's scope covered a portion of the platform, where Vidya wrote production Scala services and mentored more than 20 Java developers. Vidya makes no claim on the parts of HealthCare.gov that other teams delivered. **How much did HealthCare.gov performance improve?** HealthCare.gov performance improved by more than 30 percent across the work Vidya's team delivered on the move from Java to Scala on AWS. Vidya cites that figure from its own engagement record rather than a published CMS benchmark. The gain came from how the code was written and not from new hardware, because Scala's Future let the checks an eligibility determination makes against outside systems run at the same time rather than one after another. **Would Vidya use Scala for this work today?** No. Vidya proposes other technologies for enterprise modernization. However, what carries forward is the discipline the language enforced. Vidya still writes business rules so the build rejects any combination the statute does not allow, still makes missing data impossible to overlook, still composes complex workflows from simple rules, and still runs API calls in parallel for better scalability. Simply put, use technologies that make it harder to write bugs. A program office should choose the programming ecosystem according to that principle. Ask us for suggestions. **How does Vidya approach high-traffic government website scaling?** Vidya measures the slowest thing an actual user waits on and fixes that one before anything else. On HealthCare.gov the busiest pages spent their time waiting on outside systems, so Vidya made those checks run at the same time and recovered the wait without asking CMS to buy capacity. Vidya then measures the result so the program office can show its stakeholders a before-and-after number, the way this work produced a figure above 30 percent. ## The Challenge The Centers for Medicare & Medicaid Services (CMS) runs HealthCare.gov as the platform for the federal health insurance Marketplace, and every open enrollment season compresses a year of demand into a few weeks. An applicant who stalls on a plan comparison page may abandon the application and go without coverage for the year, and an inablilty to connect to partner APIs at the Social Security Administration or Internal Revenue Service could delay access to affordable coverage. CMS carried that seasonal surge on a conventional Java monolith running Spring, a common architecture but one that needed a boost. While the scale and seasonal load challenges for HealthCare.gov are obvious, what is less obvious is the challenge in modeling the business rules of a complex law like the Affordable Care Act into code. There is a programming maxim, "Make impossible states impossible." The code required a programming language and coding techniques that provide a safety net preventing business rule violations that would either improperly grant eligibility to applicants to the wrong resources or unfairly deny eligibility to applicants who had earned the right benefits. Accenture Federal Services held the prime relationship with CMS and brought Vidya in as a subcontractor to transition a portion of the platform from Java to Scala. As a rare vendor with prior knowledge of Scala and Play Framework, Vidya found the challenge had two dimensions. It was far more than Scala development. It was mentorship for teams of Java engineers in the idioms, design decisions, and best practices in a completely different and quite challenging programming language. ## Vidya's Approach Scala runs on the Java Virtual Machine, so a Play Framework application keeps the runtime. However, Vidya transformed the programming model with a new language and a new way of thinking about building products. HealthCare.gov applicants seeking affordable health insurance are all different. The Affordable Care Act's formulas take income, household size, prior coverage, and so much other data as inputs, but not all applicants have all that data. Missing input is an ordinary business case rather than a malfunction. The legacy Java code had no way to distinguish valid missing values from invalid gaps, so a gap in someone's record traveled quietly through the system until a later calculation reached for it and the application broke. Every one of those failures billed CMS three times over. The applicant abandoned the session or called for help. A caseworker then spent time on a problem the software should have handled. An engineer spent hours recreating a crash that left almost no trace of which record caused it. Scala has a language feature called `Option`, which forces developers to account for data that may or not be there, and Vidya made sure all the teams knew how to use it. Each eligibility formula now states which of its inputs can be absent. The build fails when any version of the code ignores one. A gap in a wage record now forces a business decision while someone is writing the code, where the fix costs minutes. This shifted quality left and prevented an entire class of Java bugs that have literally cost [billions over decades](https://www.infoq.com/presentations/Null-References-The-Billion-Dollar-Mistake-Tony-Hoare/). Vidya argued this in [Know Your Options](https://www.vidyasource.com/blog/know-your-options-scala-java/) years before the engagement, and HealthCare.gov is where the argument stopped being theoretical. Eligibility is HealthCare.gov is a function of formulas and algorithms, and CMS carries the consequence whenever code combines them into a result the statute does not allow. Vidya modeled each rule so that it declares up front what it depends on and how it can fail, which lets the build reject any combination the law does not permit. That is what "make impossible states impossible" means in practice. Write code that makes it impossible to break business rules. A rule violation caught during development costs an afternoon of engineering time. The same violation reaching an applicant costs a wrong determination, an appeal, and a program office explaining itself to an oversight body. Open enrollment hands CMS a fixed deadline and a traffic spike. A single eligibility determination has to check several outside systems. The Java code asked them one at a time, so the applicant absorbed every wait end to end. Vidya used a Scala language feature called `Future`, which lets those checks run at the same time across processors to make the most of CMS servers. Scalability improvements allowed CMS to handle the enrollment surge without buying more servers to do it, and Vidya covers this technique in [Nine Reasons to Try Scala](https://www.vidyasource.com/tutorials/nine-reasons-to-try-scala/). Engineers on the client team learned all of this on a live federal system, a harder classroom than any training course. The challenge was especially difficult because the safety and scalability features in Scala come at a price, a complex language that redefines the mental model and common paradigms in Java. V idya mentored more than 20 Java developers through a new design strategy that involved writing values that no other part of the system can quietly change and building complex business rules from simple onesones. With time, those engineers were able to extend and maintain the code themselves. ## What Changed As a result of Vidya's contribution, HealthCare.gov that ran more than 30 percent faster than the legacy Java application it replaced on infrastructure its operations staff already supported. That margin matters most in the weeks when scaling a high-traffic government website stops being an architecture debate and turns into an enrollment deadline. A shopper who loads a plan comparison quickly stays in the flow and finishes the application, and every application converted online is one that never reaches a call center. The subtler result is fewer bugs because the platform makes it harder to write bugs. Eligibility logic that once depended on a developer remembering to check for a missing value now fails to build when a case goes unhandled, so someone solves it at a desk in minutes rather than through an outage, a help desk queue, and a late night of debugging in the middle of open enrollment. Business rules are expressed with language features rather than clever programming. Maybe most importantly, former Java engineers walked away able to understand and extend the code without Vidya in the room. The approach represents our modernization strategy. A program office ready to test that can name its single busiest workflow, and Vidya's [legacy system modernization](https://www.vidyasource.com/consulting/legacy-system-modernization/) and [government modernization](https://www.vidyasource.com/consulting/government-modernization/) practices will deliver a measured before-and-after on that workflow inside one quarter. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "TRSS Turns Threat Detection Into a Named Product Line" description: "Thomson Reuters Special Services grew threat detection into a product line. A machine learning threat detection case study on Kafka, Spark, and Scala." canonical: https://www.vidyasource.com/case-studies/machine-learning-threat-detection-trss/ type: caseStudy tags: ["Machine Learning", "Data", "Architecture", "AI", "Scala", "Python", "Apache Kafka", "Apache Spark", "Spark MLlib", "MongoDB", "Play Framework", "Spring Integration", "Akka", "Angular", "Microsoft Azure"] image: https://www.vidyasource.com/img/blog/big-data.jpg --- # TRSS Turns Threat Detection Into a Named Product Line > Thomson Reuters Special Services grew threat detection into a product line. A machine learning threat detection case study on Kafka, Spark, and Scala. - Canonical page: https://www.vidyasource.com/case-studies/machine-learning-threat-detection-trss/ - Structured data (JSON-LD): https://www.vidyasource.com/case-studies/machine-learning-threat-detection-trss.json - Topics: Machine Learning, Data, Architecture, AI, Scala, Python, Apache Kafka, Apache Spark, Spark MLlib, MongoDB, Play Framework, Spring Integration, Akka, Angular, Microsoft Azure - Client: Thomson Reuters Special Services (TRSS) - Sector: National Security & Federal Law Enforcement - Engagement: Multi-year engagement - Technologies: Scala, Python, Apache Kafka, Apache Spark, Spark MLlib, MongoDB, Play Framework, Spring Integration, Akka, Angular, Microsoft Azure ## Results - **Threat detection became a TRSS product line.** TRSS sells RAPID® today as real-time risk scoring and fraud identification with entity and network analysis, inside the Fraud Detection & Program Integrity and Risk Detection capability areas alongside Thomson Reuters CLEAR®. - **12% year-over-year sales growth.** TRSS reported sales growth above 12% year over year as the threat-detection model improved after Vidya rebuilt the platform. - **Scores ready before the analyst asks.** An analyst opens the dashboard to an answer already waiting, because Vidya's platform scores each threat continuously as new data arrives and saves the finished result instead of computing it on the spot. ## Frequently asked questions **What did Vidya build for Thomson Reuters Special Services?** Vidya joined a team of senior engineers on the TRSS threat-detection application and re-architected it into a streaming machine learning pipeline. Spring Integration normalized data from multiple social media and online APIs and published it to Apache Kafka. A Scala model on Apache Spark and Spark MLlib applied natural language processing to that stream and wrote threat scores into MongoDB for the Play Framework interface to serve. **How does a real-time entity resolution and fraud detection pipeline stay fast under load?** The pipeline moves the analytical work off the request path. Vidya normalized every incoming feed into one message format, published it to a Kafka topic, and let the model score continuously against that stream. The user interface then reads a materialized view, which means response time depends on a lookup rather than on how much data arrived that morning. **Why Kafka and Spark instead of the Lambda Architecture with Hadoop?** Vidya published an early plan to evolve the platform toward Nathan Marz's Lambda Architecture with Hadoop and Elasticsearch, then changed course. Lambda asks a team to maintain a batch layer and a speed layer that each implement the same scoring rules, and TRSS engineers would have carried that duplication for the life of the product. Kafka and Spark gave them one code path, at the cost of replaying the log to recompute history. **Does Vidya build big data risk scoring platforms for federal customers today?** Yes. The TRSS engagement is Vidya's machine learning threat-detection reference, and Vidya has carried the same event-driven data architecture into federal work, including the predictive analytics platform on Azure Databricks that Vidya architected under Consular Systems Modernization at the Department of State. Vidya holds GSA MAS contract 47QTCA24D0069 for that work. ## The Challenge Thomson Reuters Special Services sold threat detection to a customer set spanning the Department of Defense, the Intelligence Community, federal law enforcement, and corporate security teams. Its software analyzed billions of public and proprietary records and turned them into intelligence an analyst could act on the same day. Vidya joined a team of senior engineers on that application, a web platform that answered an analyst's questions out of a single database. Every hour the software spent catching up to a feed was an hour a customer spent making decisions without it. Performance governed every decision. Analysts asked questions about people and networks under time pressure, and every answer came out of a database that had to do the analytical work while the analyst waited. Vidya's team tuned that database and the application hard, and the work bought real headroom. The design still tied every answer to the moment someone asked for it. That ceiling had a commercial consequence. TRSS sold this platform to commercial and government customers, so the volume of social media and online feeds the application could absorb set a practical limit on what the company could sell. ## Vidya's Approach Vidya moved the analytical work off the moment of the question. Instead of a database computing a score while an analyst waited, the new design scored each subject continuously as fresh data arrived and saved the finished result for the interface to read. The platform pulled from many social media and online feeds, folded them into one common format, and let a machine learning model score that stream around the clock. By the time an analyst opened the dashboard, the answer was already there. Vidya had announced a different plan early on and then walked away from it. The first roadmap would have split the scoring rules across two separate systems, one for live data and one for history, and TRSS engineers would have maintained both copies for the life of the product. The design Vidya shipped kept the scoring rules in one place instead. The tradeoff was that rebuilding history meant replaying the stream rather than rerunning a batch job, and on a platform whose value comes from freshness, that landed on the right side of the trade. The engineering choices underneath all of this served one business need: a threat score a customer can trust. Vidya built the platform so the same input always produces the same score, every time it runs. That consistency is what lets TRSS defend a score when a customer asks how the system reached it, and it is what keeps a result explainable a year after the fact. Vidya had argued for this discipline publicly in [Java Is Dysfunctional With Big Data](https://www.vidyasource.com/blog/java-is-dysfunctional-with-big-data/) and returned to it years later in [The Business Case for Functional Programming](https://www.vidyasource.com/blog/business-case-for-functional-programming/). The same reasoning still shapes how Vidya approaches [AI consulting](https://www.vidyasource.com/consulting/ai-consulting/) engagements. One boundary deserves naming. Vidya's team built the ingestion pipeline, the machine learning plumbing, and the interface that surfaced scores. TRSS analysts owned the tradecraft, the proprietary record sources, and the definition of what counts as a threat. The product TRSS sells today carries years of TRSS investment beyond anything a single engagement contributed. ## What Changed TRSS analysts stopped waiting on a database to compute an answer and started reading a score the system had already produced. TRSS reported sales growth above 12% year over year as the model improved. The threat-detection platform Vidya helped build grew into a core TRSS business offering, and the company sells RAPID® today as real-time risk scoring and fraud identification with entity and network analysis, inside its Fraud Detection & Program Integrity and Risk Detection capability areas alongside Thomson Reuters CLEAR®. The move generalizes to any risk scoring platform that still makes users wait while it does the math. Organizations in that position gain a system that scores continuously in the background and an interface that simply reads the finished answer. Their scoring rules then live in exactly one place, which is what keeps a score explainable a year later. Vidya delivers that first working version within a quarter, and the [software architecture](https://www.vidyasource.com/consulting/software-architecture/) practice that grew out of this work carries the pattern into federal and commercial programs today. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Tripling Reservations on Recreation.gov" description: "Recreation.gov tripled reservations on a Go and React microservices platform. A federal modernization case study on Vidya's engineering under Booz Allen Hamilton." canonical: https://www.vidyasource.com/case-studies/recreation-gov-modernization/ type: caseStudy tags: ["Modernization", "Government", "Architecture", "Go", "React", "Cloud", "DevOps", "Go", "Go kit", "React", "Redux", "Cassandra", "AWS", "Docker", "Kubernetes", "Jenkins"] image: https://www.vidyasource.com/img/blog/recreation.png --- # Tripling Reservations on Recreation.gov > Recreation.gov tripled reservations on a Go and React microservices platform. A federal modernization case study on Vidya's engineering under Booz Allen Hamilton. - Canonical page: https://www.vidyasource.com/case-studies/recreation-gov-modernization/ - Structured data (JSON-LD): https://www.vidyasource.com/case-studies/recreation-gov-modernization.json - Topics: Modernization, Government, Architecture, Go, React, Cloud, DevOps, Go, Go kit, React, Redux, Cassandra, AWS, Docker, Kubernetes, Jenkins - Client: Recreation.gov - Sector: Federal Government - Engagement: Six-month engagement - Technologies: Go, Go kit, React, Redux, Cassandra, AWS, Docker, Kubernetes, Jenkins ## Results - **Reservations tripled.** Recreation.gov tripled its reservations after the platform moved to a microservices architecture, so far more visitors claimed campsites, permits, and tours across federal parks, forests, and public lands. - **Large revenue increases for public lands.** Recreation.gov turned that larger volume of reservations into large revenue increases for the federal land management agencies whose sites it serves, money that funds the upkeep of the places visitors came to see. - **A broken build caught in minutes.** Recreation.gov ran every code change through an automated pipeline on Jenkins, Docker, and Kubernetes, watched by a dedicated Site Reliability Engineering team, so engineers caught a broken build within minutes instead of during a release. ## Frequently asked questions **What was Vidya's role in the Recreation.gov modernization?** Vidya worked as a subcontractor to Booz Allen Hamilton, the prime contractor on the Recreation One Stop (R1S) program. Vidya developed Go APIs and React components and modeled data in Cassandra on Amazon Web Services (AWS). Booz Allen Hamilton and the federal government led the program and own the overall outcome, and Vidya delivered a defined portion of the engineering underneath it. **What technology stack does Recreation.gov use?** Recreation.gov runs one of the more ambitious customer-facing stacks in the federal government. Go powers its microservices through the Go kit framework, React with Redux and React Router drives the interface, and Cassandra on AWS stores the data. Jenkins, Docker, and Kubernetes carry each code change to production under a dedicated Site Reliability Engineering team. **How did the modernization help Recreation.gov triple reservations?** Recreation.gov built its microservices architecture for performance first, because thousands of visitors compete at once for high-demand permits and lotteries at places like the Grand Canyon and Mount Whitney. A platform that stays fast and available under that load keeps people in the flow long enough to finish a booking. Recreation.gov tripled its reservations as it scaled, and Vidya cites that figure from its own engagement record for the portion it delivered rather than a published program benchmark. **How does Vidya approach high-traffic platform modernization?** Vidya decomposes the platform into small services that one team can own end to end, then puts continuous delivery underneath them so a change reaches production the day an engineer writes it. Vidya reserves the newest tooling for the places it earns its risk and leans on proven runtimes everywhere else. Vidya then measures the result the program office can show its stakeholders, the way this engagement produced a reservation figure that tripled. ## The Challenge Recreation.gov is the federal government's single front door for booking the outdoors. A visitor reserves a campsite, claims a backcountry permit, books a tour, or enters a lottery for the most sought-after trips, all across the parks, forests, waterways, and monuments that federal land management agencies steward. The Recreation One Stop (R1S) program runs that platform for the public. People call it the "Airbnb for Camping," and the comparison sets the expectation that arrives with it. That expectation is the hard part. Thousands of visitors converge on the same release window for a high-demand permit at the Grand Canyon or a summit lottery at Mount Whitney, and they arrive all at once. A platform that stalls under that surge loses the booking, and the visitor loses the trip. Recreation.gov needed an architecture built for performance and scale before it could be built for anything else. Booz Allen Hamilton held the prime contract on the R1S program and brought Vidya in as a subcontractor. Vidya joined a team already committed to a modern, customer-facing stack, and Vidya's scope covered a defined portion of the engineering rather than the whole platform. ## Vidya's Approach Vidya built for the surge from the first commit. Vidya developed Go APIs on the Go kit microservices framework, so each capability ran as a small service the platform could scale on its own rather than as one monolith that rose or fell together. Vidya modeled the data in Cassandra on AWS, a store designed to stay available and fast when write traffic spikes against it. A visitor racing thousands of others for the same permit feels that decision directly. Vidya built the interface as custom React components with Redux and React Router, so the booking flow stayed responsive while the services behind it did the heavy work. The front end and the services met at clear REST API contracts, which let the interface team and the API team move on separate schedules without waiting on each other. That separation matters most in the weeks before a marquee lottery opens, when both sides ship changes under a fixed public deadline. Recreation.gov treated delivery itself as an engineering product, and Vidya worked inside that discipline. Every code change ran through an automated pipeline that built the Go services on Jenkins, packaged them as Docker images, and let Kubernetes orchestrate them, all under a dedicated Site Reliability Engineering team. A broken build surfaced within minutes of the change that caused it. The program invested more in continuous delivery than many efforts spend on features, and that investment kept a public, revenue-bearing platform releasing safely while demand climbed. ## What Changed Recreation.gov tripled its reservations as the modernized platform scaled to meet demand, and more visitors reached the campsites, permits, and tours they came for. That larger volume of reservations turned into large revenue increases for the federal land management agencies whose sites the platform serves, money that funds the upkeep of the places people booked. A visitor who once bounced off a stalled page now finishes the reservation, and every booking completed online is one the agencies never work by hand. The method carries to any high-traffic platform a program office is afraid to change under load. Vidya decomposes the system into small services one team can own, puts continuous delivery beneath them so a change ships the day it is written, and measures the result the office can take to its stakeholders. Name the single busiest booking or transaction flow your platform runs, and Vidya will return a target microservices architecture and a service-by-service migration plan for it within one quarter. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Neustar Modernizes Telecom Billing Behind a REST API" description: "Neustar modernized legacy telecom billing into a Java REST API on Dropwizard and Amazon EC2. How Vidya handled a legacy API migration to REST." canonical: https://www.vidyasource.com/case-studies/api-modernization-neustar/ type: caseStudy tags: ["Architecture", "Cloud", "Java", "Modernization", "Java", "Groovy", "REST", "Dropwizard", "Gradle", "Jenkins", "AWS", "Amazon EC2"] image: https://www.vidyasource.com/img/blog/technology.jpg --- # Neustar Modernizes Telecom Billing Behind a REST API > Neustar modernized legacy telecom billing into a Java REST API on Dropwizard and Amazon EC2. How Vidya handled a legacy API migration to REST. - Canonical page: https://www.vidyasource.com/case-studies/api-modernization-neustar/ - Structured data (JSON-LD): https://www.vidyasource.com/case-studies/api-modernization-neustar.json - Topics: Architecture, Cloud, Java, Modernization, Java, Groovy, REST, Dropwizard, Gradle, Jenkins, AWS, Amazon EC2 - Client: Neustar - Sector: Commercial - Engagement: Three-year engagement - Technologies: Java, Groovy, REST, Dropwizard, Gradle, Jenkins, AWS, Amazon EC2 ## Results - **Billing became a service any team could call.** Neustar's product teams metered and charged for usage by calling one billing service from any language and any platform in the NexGen cloud portfolio. - **A broken build caught in minutes.** Every code change ran through an automated build and test, so Neustar's engineers caught a broken build within minutes of the change that caused it instead of during a release. - **NexGen cloud portfolio gained a shared service.** Neustar formed the NexGen group to expand its cloud offerings, and the billing API gave every new offering in that portfolio a metering and charging path it did not have to build itself. ## Frequently asked questions **What did Vidya actually modernize at Neustar?** Vidya joined a team of senior engineers inside Neustar's NexGen group and rebuilt the company's billing capability as a REST API in Java and Groovy on Dropwizard, deployed to Amazon EC2. Gradle handled project automation and Jenkins handled continuous integration. The engagement ran roughly three years. **How do you run a legacy API migration to REST without breaking downstream consumers?** Vidya publishes the new HTTP contract before the implementation moves, so consuming teams code against a stable surface while the legacy system still answers underneath. Reserving GET for reads keeps those calls idempotent and cacheable, which means a client can retry a failed request safely. Consumers then migrate on their own schedule, and no one has to survive a single cutover date. **Why Dropwizard instead of a full application server?** Dropwizard packages an HTTP service and its embedded server into a single executable JAR file. Neustar's team shipped one artifact to an Amazon EC2 instance and skipped the shared application server entirely, which removed a class of container configuration failures from the release path. Each service also owned its own runtime, so one team's upgrade never forced an upgrade on another team. **Does Vidya do telecom API platform modernization outside government work?** Yes. Neustar is Vidya's commercial API modernization reference, and Vidya has carried the same HTTP and contract discipline into federal programs including Consular Systems Modernization at the Department of State and HealthCare.gov at the Centers for Medicare and Medicaid Services. ## The Challenge Every phone call, fax, and computer connection in North America depended on infrastructure Neustar ran. The company had rescued the ten-digit telephone numbering plan with a solution the FCC mandated, and every telephone company on the continent held a physical interface into Neustar's directory system. Carriers treated that directory the way banks treat a settlement ledger. Neustar had earned the right to be boring. Neustar's ambitions had moved past the directory by then. The company sold cloud services to enterprises and stood up a group called NexGen to expand that portfolio across new technologies and platforms. Billing stood in the way. Every cloud offering Neustar wanted to launch had to meter usage and charge for it, and the company's billing capability predated the portfolio it now had to serve. Billing also tolerates less error than most software. A wrong charge lands on a carrier's accounts payable desk and turns into a dispute. A missed charge never comes back as revenue. Neustar needed an interface that teams across the company could call over plain HTTP, and it needed the existing billing behavior to survive the move intact. ## Vidya's Approach Vidya joined the NexGen group as part of a team of senior engineers and rebuilt billing as a service any product could call over plain web requests. The team packaged that service so it shipped as a single self-contained unit rather than something an operator had to install into shared server software. Neustar's operators gained a release path with one moving part and one fewer class of failures on the way to production. Neustar's product teams would depend on whatever design shipped for years, so Vidya weighed every decision with that in mind. The team designed the service so a client could safely retry a failed request without charging anyone twice, which matters most on the day a network hiccups in the middle of a transaction. Money got the same scrutiny. Halfway through the engagement, Vidya published [Zero Tolerance](https://www.vidyasource.com/blog/zero-tolerance-java/) on a subtle way that routine money math in Java quietly returns the wrong answer. A billing service cannot ship a bug that charges the wrong amount. That same discipline still anchors Vidya's [software architecture](https://www.vidyasource.com/consulting/software-architecture/) practice. Vidya also chose the deployment approach Neustar's staff already ran every day rather than a container technology that had only just shipped its first stable release. A revenue-bearing service is the wrong place to prove out an immature runtime. Every code change then ran through an automated build and test, and a broken build surfaced within minutes of the change that caused it, so Neustar's engineers kept their attention on billing rules while the release mechanics ran themselves. Neustar's own engineers owned the number directory and the carrier interfaces into it, and Vidya's team never touched them. Vidya's scope stayed inside the NexGen group's cloud portfolio. That boundary is worth naming, because a national number portability directory and a billing API are different engineering problems with different consequences when they fail. ## What Changed Neustar's product teams reached billing with a plain web request from any language and any platform in the cloud portfolio, which meant a new offering could meter and charge for usage without standing up a billing project of its own. The engagement ran about three years, long enough for Vidya to carry the API past its first release and into the unglamorous work of keeping a revenue-bearing service healthy on Amazon EC2. Neustar got a billing capability its roadmap could depend on. The sequence still holds for any team modernizing a system other groups depend on. Vidya makes the old capability callable behind a clear, stable interface the consuming teams can build against, then keeps each piece small enough for one team to own end to end. The legacy implementation moves behind that contract on a schedule the business picks. Organizations running a revenue system they are afraid to change can start that sequence inside a single quarter, and Vidya runs it today across commercial platforms and federal [legacy system modernization](https://www.vidyasource.com/consulting/legacy-system-modernization/) programs alike. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "The Curriculum That Became the Federal DITAP Program" description: "The Executive Office of the President got a digital IT acquisition curriculum that became DITAP, the credential federal software buyers now have to earn." canonical: https://www.vidyasource.com/case-studies/ditap-digital-acquisition-training-eop/ type: caseStudy tags: ["Government", "Project Management", "Modernization", "Culture"] image: https://www.vidyasource.com/img/blog/class.jpg --- # The Curriculum That Became the Federal DITAP Program > The Executive Office of the President got a digital IT acquisition curriculum that became DITAP, the credential federal software buyers now have to earn. - Canonical page: https://www.vidyasource.com/case-studies/ditap-digital-acquisition-training-eop/ - Structured data (JSON-LD): https://www.vidyasource.com/case-studies/ditap-digital-acquisition-training-eop.json - Topics: Government, Project Management, Modernization, Culture - Client: Executive Office of the President - Sector: Federal Government - Engagement: Federal training engagement ## Results - **DITAP became a required federal credential.** Contracting officers who hold FAC-C (Professional) and work digital services acquisitions above the FAR 13.500(c) thresholds must now complete DITAP. - **25 to 30 students per cohort.** The TechFAR Hub runs DITAP cohorts of 25 to 30 students over roughly six months, so each cycle moves a whole working group through the material at once. - **60 to 80 Continuous Learning Points.** Graduates earn 60 to 80 Continuous Learning Points and a DITAP Training Certificate, which qualifies them for the Digital Services Credential through the Federal Acquisition Institute. - **An OFPP memo formalized the specialization.** The Office of Federal Procurement Policy established the Federal Acquisition Certification in Core-Plus Specialization in Digital Services in a memo, which gave the training a permanent place in the federal certification structure. ## Frequently asked questions **What is the origin of DITAP training?** The Executive Office of the President competed a training effort aimed at the federal acquisition workforce. Vidya earned a place on the winning team and worked with the United States Digital Service, 18F, and other vendors to create the first iteration of the Digital IT Acquisition Professional curriculum. The TechFAR Hub runs that program today as DITAP. **What did Vidya actually do on the DITAP program?** Vidya teamed with other vendors alongside the United States Digital Service and 18F to build the first iteration of the curriculum, and Vidya taught the course for the Executive Office of the President. Vidya held a teaming position on that award and never held the prime contract. The government owns the curriculum, and the program now runs without any of the original vendors. **Is DITAP required for federal contracting officers?** Yes, for a defined population. An Office of Federal Procurement Policy memo established the Federal Acquisition Certification in Core-Plus Specialization in Digital Services. FAC-C (Professional) holders working digital services acquisitions above the FAR 13.500(c) thresholds have to complete DITAP, and graduates earn 60 to 80 Continuous Learning Points toward it. **Does Vidya still work on federal software acquisition training and strategy?** Yes. Vidya belongs to the GAO Agile Expert Group, has contributed to Government Accountability Office agile reports, and carries the same position on contract structure into federal market research today. Vidya also runs engineering and agile training courses for government and commercial teams. ## The Challenge Federal agencies once bought custom software the way they bought office buildings. A contracting officer wrote the requirements up front, competed a fixed scope, and held the vendor to a schedule of phase gates and document deliverables. That structure suits construction. It suits software badly, because a requirements document written two years before launch describes a product nobody has used, and the contract then punishes the vendor who learns something in month nine. The acquisition workforce carried no training that offered an alternative. Contracting officers across the government had spent careers mastering the Federal Acquisition Regulation, and none of their professional development explained how to write a statement of work that pays for working software every few weeks. Agencies kept buying waterfall because waterfall was the only shape their buyers knew how to price, evaluate, and defend. The Executive Office of the President moved on that gap. The United States Digital Service and 18F had started rotating engineers out of industry and into agencies, and those technologists kept landing inside programs whose contracts made iterative delivery impossible in practice. The Office of Management and Budget competed a training effort aimed squarely at the people who write the contracts. ## Vidya's Approach Vidya earned a place on the winning team for that competition and worked alongside the United States Digital Service and 18F, together with other vendors, to build the first iteration of what became the Digital IT Acquisition Professional program. Vidya authored curriculum and taught the course for the Executive Office of the President. The goal was narrow and concrete, namely teaching federal procurement officials to write contracts that motivate better vendor behavior and therefore better software. Contracting officers filled the room, and that audience decided nearly every design choice in the material. The team drew a hard boundary around scope. The curriculum would cover contract structure, market research, and vendor incentives. It would leave software engineering to the engineers. A contracting officer who understands why a fixed-scope award produces an unusable product will restructure the next acquisition, and that single change buys an agency more than any engineering vocabulary the same officer could memorize. Vidya still argues the position that curriculum started from. Any contract type that fixes product scope up front hands the program a bad outcome no amount of oversight recovers, and confidence in one implementation should never harden into lock-in to the vendor who built it. Vidya carries that argument into federal market research today and laid out the underlying history publicly in [Don't Go Chasing Waterfall](https://www.vidyasource.com/blog/dont-go-chasing-waterfall/). Buying behavior follows organizational culture, which is the same problem Vidya works on inside agencies through its [technology culture](https://www.vidyasource.com/consulting/technology-culture/) practice. One limit belongs on the record. Vidya held a teaming position on that award rather than the prime contract, and the government owns the curriculum outright. Vidya kept the thread going through membership in the GAO Agile Expert Group and contributions to Government Accountability Office agile reports, which is where the federal conversation about measuring agile delivery continued after the course ended. ## What Changed Federal contracting officers now hold a credential for this work. The TechFAR Hub runs [DITAP](https://techfarhub.usds.gov/get-started/ditap/) as a standing program with cohorts of 25 to 30 students over roughly six months, covering digital services market intelligence, stakeholder analysis, and leading change as a digital IT acquisition professional. Graduates earn 60 to 80 Continuous Learning Points and a DITAP Training Certificate that qualifies them for the Digital Services Credential through the Federal Acquisition Institute. An Office of Federal Procurement Policy memo established the Federal Acquisition Certification in Core-Plus Specialization in Digital Services, and FAC-C (Professional) holders who work digital services acquisitions above the FAR 13.500(c) thresholds have to take the training. A course agencies once volunteered for became a requirement their senior buyers cannot route around. An agency that wants the delivery behavior DITAP describes still has to write the solicitation that pays for it, and that work happens one acquisition at a time. Vidya brings the same argument to [government modernization](https://www.vidyasource.com/consulting/government-modernization/) engagements, where the contract shape and the architecture get decided in the same room. Program offices planning a digital services acquisition can put that structure in place before the solicitation goes out, and the vendors who answer it will build differently because of it. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Modernizing Federal Technical Hiring at the White House" description: "Mediabarn brought Vidya to White House Technology Day, where federal recruiters got a working playbook for hiring elite government software engineers." canonical: https://www.vidyasource.com/case-studies/federal-tech-hiring-white-house-mediabarn/ type: caseStudy tags: ["Government", "Partners", "Project Management", "Culture"] image: https://www.vidyasource.com/img/blog/tech-hiring.png --- # Modernizing Federal Technical Hiring at the White House > Mediabarn brought Vidya to White House Technology Day, where federal recruiters got a working playbook for hiring elite government software engineers. - Canonical page: https://www.vidyasource.com/case-studies/federal-tech-hiring-white-house-mediabarn/ - Structured data (JSON-LD): https://www.vidyasource.com/case-studies/federal-tech-hiring-white-house-mediabarn.json - Topics: Government, Partners, Project Management, Culture - Client: Mediabarn - Sector: Federal Government - Engagement: Conference engagement ## Results - **Federal recruiter cohort briefed at the White House.** Vidya's founder delivered "Tech Landscape and Government Realities: Modernizing Government Hiring" on Technology Day at the White House, an engagement Vidya performed for Mediabarn. - **A recruiter gained language for his leadership.** One attendee said he had held similar ideas for a while but had lacked the vocabulary to present them to his agency's leadership until he saw the presentation. - **Engineering roles mapped to real product categories.** The talk paired software product categories that industry and government both build, from websites to machine learning pipelines to AI, with the roles that deliver each one. ## Frequently asked questions **What was Vidya's role in the White House engagement for Mediabarn?** Vidya performed this work for Mediabarn as an advisory and thought-leadership engagement. The White House hosted Technology Day and supplied the audience, a cohort of federal recruiters. Vidya's founder wrote and delivered the talk, and Vidya shipped no software on this engagement. **How does a federal agency modernize technical hiring?** Start with the artifacts a candidate actually reads. Align position titles and career ladders with the language private industry uses, so an engineer can tell what the job involves and where it leads. Then replace the coding interview with a structured conversation about real work the candidate has done. Federal technical hiring modernization fails when an agency copies commercial practice wholesale, because commercial practice still runs on the screen that costs agencies the candidates they want. **Why does Vidya argue against coding interviews for government software engineer recruiting?** Coding interviews measure short-term recall of computer science trivia and composure under observation. A North Carolina State University and Microsoft study found that every woman who took a public technical interview failed, while every woman who took the same interview privately passed. Agencies competing for White House tech talent cannot afford a filter that removes qualified engineers for reasons unrelated to the work. **Which hiring change should a program office make first?** Rewrite one position description and the interview loop attached to it, then run that loop on the next open requisition. A single requisition gives the hiring manager a controlled comparison against the agency's standard process within one hiring cycle. Vidya recommends measuring the change by who applies and who accepts, because those two numbers move before any org-wide policy does. ## The Challenge Federal agencies spent decades outsourcing software product development to private vendors, and they eventually wanted that subject matter expertise back in house. Hiring stood in the way. Government job titles and career ladders mapped poorly onto what an engineer reads on a private-industry posting, so a strong candidate could not tell what the work involved or where it led. Agencies competed for attention against Big Tech at the same time, and a segment of elite talent regards Uber, Meta, Spotify, and Google as the lifelong dream job. Recruiters do not argue a dream away. The industry playbook those agencies wanted to borrow carried its own defect. Commercial software hiring runs on the coding interview, an exercise that asks a candidate to whiteboard a doubly-linked list or a merge sort while a panel watches. A North Carolina State University and Microsoft study found that every woman who took a public technical interview failed, while every woman who took the same interview privately passed. An agency importing commercial practice wholesale would import that screen along with it and keep losing the engineers it set out to recruit. Mediabarn engaged Vidya to make the case to the people who could act on it. The White House hosted Technology Day, and a cohort of federal recruiters made up the audience. Vidya's founder delivered a talk titled "Tech Landscape and Government Realities: Modernizing Government Hiring." ## Vidya's Approach Vidya delivered advice on this engagement and wrote no code, so the recommendations carry the entire value and have to hold up on their own. Recruiters win by choosing their ground. Vidya advised the cohort to stop spending effort on engineers who have already settled on Big Tech and to concentrate on the sizable population of elite talent that wants mission work. Those engineers understand that disabled people, elderly people, and veterans carrying the consequences of war depend on federal services that function. Agencies had meanwhile begun realigning job titles and career paths with private-industry norms, and Vidya showed examples of government leaders already doing it, because a posting a candidate can read is the cheapest fix on the list. The interview itself needs replacing. Vidya recommends a structured conversation in which the interviewer opens by stating business goals, mission, and role expectations, then works through six questions. Ask candidates to walk through problems they solved at work, and follow those threads until the technical judgment shows. Ask what support they need to do their best work, which moves the burden onto the employer where it belongs. Vidya lays out the full set and the evidence behind it in [why coding interviews are the worst](https://www.vidyasource.com/blog/why-coding-interviews-are-the-worst/). Teams that cannot abandon coding exercises have three humane adaptations available. Offer a choice between live coding and a short paid take-home. Hand candidates a real but non-critical problem drawn from actual work. Collaborate through the exercise with access to normal resources, including AI. Culture decides whether the hire stays. Vidya described flow state as the foundation of good engineering work and showed how closely it hews to agile principles, then placed responsibility for creating flow state on management, citing Sarah Drasner's argument that flow matters more than passion. Screening for passion through side projects and weekend commits filters for free time, and free time is a privilege. That same reasoning drives Vidya's position that systems and leadership produce team results, which [how sports busts a common myth about software engineering teams](https://www.vidyasource.com/blog/how-sports-busts-common-myth-software-engineering-teams/) develops through decades of roster-building failures. Vidya closed by mapping product categories that industry and government both build, from websites to machine learning pipelines to AI, onto the roles suited to deliver each one, because a program office staffing an AI product needs a different mix of engineers than one staffing a website. One limit deserves a plain statement. Industry has not mastered hiring in its own right, so no finished commercial template existed to hand the recruiters. Vidya argued instead for practices most commercial teams still ignore. Vidya also reports no hires made and no policy changed, because the downstream effect of a talk resists measurement once the applause fades. ## What Changed Federal recruiters left the session with language for a case many of them already believed. One attendee said afterward that he had held similar ideas himself but had lacked the vocabulary to present them to his agency's leadership until he saw the presentation, and he left planning forward-looking changes to his own hiring process. That reaction is the honest measure available, and government software engineer recruiting changes one hiring manager at a time. An agency ready to move on this can start with a single requisition. Vidya's [technology culture](https://www.vidyasource.com/consulting/technology-culture/) practice rewrites one position description and the interview loop attached to it, then runs the revised loop with the hiring manager on the next open role, so the program office compares applicant quality and acceptance rate against its standard process within one hiring cycle. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Nina Day Casts Global Brands from One Talent Database" description: "Nina Day casts for Adidas, Chanel, and the United Nations on a talent database Vidya built in Scala, Play Framework, and MongoDB. A casting platform case study." canonical: https://www.vidyasource.com/case-studies/casting-platform-nina-day/ type: caseStudy tags: ["Architecture", "Modernization", "Partners", "Security", "Scala", "Play Framework", "MongoDB", "React", "TypeScript", "Alpakka", "Amazon S3", "Amazon CloudFront", "Cloudinary", "Flowplayer", "Ruby on Rails", "PostgreSQL"] image: https://www.vidyasource.com/img/partners/ninaday-social.jpg --- # Nina Day Casts Global Brands from One Talent Database > Nina Day casts for Adidas, Chanel, and the United Nations on a talent database Vidya built in Scala, Play Framework, and MongoDB. A casting platform case study. - Canonical page: https://www.vidyasource.com/case-studies/casting-platform-nina-day/ - Structured data (JSON-LD): https://www.vidyasource.com/case-studies/casting-platform-nina-day.json - Topics: Architecture, Modernization, Partners, Security, Scala, Play Framework, MongoDB, React, TypeScript, Alpakka, Amazon S3, Amazon CloudFront, Cloudinary, Flowplayer, Ruby on Rails, PostgreSQL - Client: Nina Day - Sector: Media & Advertising - Engagement: Ongoing engagement - Technologies: Scala, Play Framework, MongoDB, React, TypeScript, Alpakka, Amazon S3, Amazon CloudFront, Cloudinary, Flowplayer, Ruby on Rails, PostgreSQL ## Results - **13 years on one platform.** Nina Day has retained Vidya on a monthly retainer for over a decade, through a full rebuild of the talent database as the agency's roster and media volume grew. - **Exceptional on quality and reliability.** Nina Day's president rated Vidya Exceptional on quality, business relations, key personnel, and reliability on the past-performance questionnaire Vidya submitted for its GSA Multiple Award Schedule contract. - **Global brand roster.** That client roster now reaches across advertising, luxury retail, and global institutions. The platform carries the search and presentation workload behind all of it. - **Talent maintain their own profiles.** A secure profile lets talent upload their own headshots, reels, and resumes, so the roster stays current without the agency typing it in. ## Frequently asked questions **What does a casting database platform actually have to do?** It has to hold rich talent attributes and large media in the same record, then return a shortlist fast enough to use inside a casting meeting. Nina Day's platform handles secure self-entry by talent, granular query across the roster, presentation building with drag-and-drop ordering, permissioned client galleries, and PDF export at the presentation, category, or individual level. Vidya built all of it on Play Framework in Scala with MongoDB. **What technology stack runs the Nina Day talent database?** Play Framework in Scala serves the application and MongoDB stores talent profiles, whose attributes differ from one performer to the next. The front end runs React and TypeScript. Amazon S3 holds the media behind Amazon CloudFront, with Cloudinary for image delivery and Flowplayer for video playback. **Does Vidya build talent management software for agencies?** Yes. Nina Day is Vidya's commercial reference for custom talent management software, and Vidya has run the platform as prime for over a decade. The same team also delivers commercial API and platform modernization, including the billing API Vidya built for Neustar. **How do you keep a custom media search platform fast when the assets are large?** Vidya separates the metadata path from the media path. Search hits MongoDB with indexing, paging, and Scala parallelism, while headshots and reels stream from object storage through a content delivery network close to the viewer. The Alpakka connector for Amazon S3 streams uploads asynchronously, which keeps the memory footprint per upload low even when several people upload reels at once. ## The Challenge Nina Day casts models, actors, and musicians for advertising campaigns, film, and television out of New York. The agency wins work by finding the one right face before anyone else finds it. That search once ran through email attachments, shared folders, and a casting director's memory of who fit a similar brief two years earlier. Every new campaign started the hunt over. The media made the problem heavier than a normal database problem. A single talent record carries headshots, a portfolio, a video reel, and a resume. Nina Day's clients open those files from wherever the campaign happens to live. A creative director who waits on a reel to buffer stops watching it. Presentations carried the commercial risk. A brand client needs a curated gallery it can browse, mark up, and circulate internally. A casting agency can never let one client's shortlist reach another. Nina Day needed permissions, ordering, and an export its clients could forward to a colleague who never logs into the platform. ## Vidya's Approach Vidya built the first talent database. Talent uploaded their own headshots, portfolios, reels, and resumes through a secure profile, which moved the data-entry cost onto the people who own the material. The platform stored that media in the cloud and served it from locations close to each viewer, so a client on another continent did not wait on a slow download. Nina Day's roster grew without the agency hiring anyone to maintain it. As submission volume grew, the first version hit a wall. Processing images, handling video, and generating presentation PDFs all run long, and the original build let that heavy work hold up everything else. Vidya rebuilt the platform so those jobs run in the background without making anyone wait, and so a large upload no longer strains the system even when several people upload reels at once. The new design also fits the shape of casting data, where one performer's profile looks nothing like the next. A casting director now gets search results back in the time a search should take. Nina Day's team lives in the browser all day, so the interface earned the same scrutiny as everything behind it. Vidya weighed the leading options and chose a foundation with a large, established community, because that let the team reuse proven building blocks instead of rewriting every screen from scratch. Vidya then layered in safeguards that catch whole categories of mistakes before they ever reach a user, so the screens the agency depends on stay predictable under daily use. That is the same tradeoff that runs through Vidya's [business case for functional programming](https://www.vidyasource.com/blog/business-case-for-functional-programming/) and its [software architecture](https://www.vidyasource.com/consulting/software-architecture/) practice. Vidya accepts a little more tooling in exchange for software that is harder to break. The commercial value sits in the presentation layer. Nina Day's team defines categories, organizes talent inside them, and reorders each performer's files by drag and drop. Brand clients open only the presentations their permissions allow, select the talent they prefer, and generate a PDF of a full presentation, one category, or a single performer. The platform also rolls selections up by reviewer and surfaces where reviewers converged, so the agency walks into the callback meeting already knowing where the room agrees. ## What Changed Nina Day's casting directors now run a query, assemble a presentation, and put it in front of a brand's creative team in one sitting. The agency's public client list runs through Adidas, Chanel, Nike, and the United Nations. It also represents photographers and directors including Martin Schoeller and Inez & Vinoodh. Nina Day won those accounts on its casting judgment. The platform absorbed the operational load that arrived with them, and a build for a New York agency still serves that roster today. Nina Day's directors still decide who has Blue Steel, and the platform gets them to that decision with the shortlist already built. Vidya has supported the platform on a monthly retainer for over a decade. Nina Day rated the work Exceptional on quality, business relations, key personnel, and reliability when it completed a past-performance questionnaire for Vidya's GSA Multiple Award Schedule contract. Agencies sitting on a decade of talent assets in shared folders can reach the same place in stages. Vidya starts with the search and the media pipeline, because those two carry the entire casting workflow, and a first working search over an existing asset library lands inside a single quarter. Vidya runs that sequence today for commercial platforms like Nina Day and [Neustar](https://www.vidyasource.com/case-studies/api-modernization-neustar/), and for federal [legacy system modernization](https://www.vidyasource.com/consulting/legacy-system-modernization/) programs. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Concurrent Real-Time Briefs Two Audiences With One White Paper" description: "Concurrent Real-Time commissioned Vidya for a recurring stream of white papers on RedHawk Linux, KVM-RT, and Missile TestBench for Golden Dome buyers." canonical: https://www.vidyasource.com/case-studies/technical-white-papers-concurrent-real-time/ type: caseStudy tags: ["Architecture", "Government", "Testing"] image: https://www.vidyasource.com/img/blog/blog-posts.jpg --- # Concurrent Real-Time Briefs Two Audiences With One White Paper > Concurrent Real-Time commissioned Vidya for a recurring stream of white papers on RedHawk Linux, KVM-RT, and Missile TestBench for Golden Dome buyers. - Canonical page: https://www.vidyasource.com/case-studies/technical-white-papers-concurrent-real-time/ - Structured data (JSON-LD): https://www.vidyasource.com/case-studies/technical-white-papers-concurrent-real-time.json - Topics: Architecture, Government, Testing - Client: Concurrent Real-Time - Sector: Defense & Real-Time Systems - Engagement: Ongoing engagement ## Results - **Every outline and draft accepted.** Concurrent Real-Time approved each outline Vidya proposed and each draft Vidya delivered across a recurring engagement, so no paper went back for a technical rewrite. - **One white paper became three deliverables.** The company expanded each commission from a single white paper into a white paper, an executive summary for the program office, and a companion article for wider distribution. - **Coverage across MTB Levels 1, 2, and 3.** Vidya wrote across the full Missile TestBench line and into subsystem validation and sensor simulation, alongside separate papers on the RedHawk Linux platform and KVM-RT. ## Frequently asked questions **What did Vidya deliver to Concurrent Real-Time?** Vidya delivered a recurring stream of technical white papers, executive summaries, and companion articles across a recurring engagement. The subject matter covered RedHawk Linux and KVM-RT. It reached the Missile TestBench solution at Levels 1, 2, and 3, along with subsystem validation and sensor simulation. Concurrent Real-Time accepted every outline and every draft, then expanded each commission from a single white paper into a three-piece package. **How do you write real-time systems technical content for engineers and program managers at the same time?** Vidya opens on the mission consequence and introduces the mechanism after the reader has a reason to care. A program manager reads what a slipped deadline costs a test campaign before meeting kernel preemption, interrupt latency, or CPU shielding. The technical reader still reaches full depth by the middle of the document. One paper then survives engineering review inside the vendor and still briefs a buyer who has never tuned a kernel. **Why hire a practicing engineer to write a technical white paper?** Vidya has architected enterprise systems for 25 years, led analytics as technical lead on the DARPA XDATA program at Sotera Defense Solutions, and served as site lead on the U.S. Army INSCOM Enterprise Platform. Vidya reads specifications and source material directly, which removes the interview-and-translate cycle a non-engineer writer needs before every draft. Concurrent Real-Time accepted every draft and kept commissioning new topics. **Does Vidya offer defense technology writing services to other companies?** Yes. Vidya writes white papers, executive summaries, and companion articles for companies selling deterministic and safety-critical systems into defense programs. Vidya worked at Sotera Defense Solutions on DARPA and U.S. Army programs and knows how these systems get evaluated and bought. Vidya delivers an outline within a week of a topic kickoff and a full draft package inside the following two weeks. ## The Challenge Concurrent Real-Time builds hard real-time computing systems, meaning systems where a late answer counts as a wrong answer. RedHawk Linux, the company's real-time Linux platform, holds interrupt response and scheduling latency inside a bounded window even when the machine is busy. KVM-RT carries that determinism into virtual machines. The Missile TestBench, or MTB, bundles those capabilities into a hardware-in-the-loop test solution, where real flight hardware runs against simulated sensors and simulated physics on a clock that never slips. A program proves a weapon that way long before it flies. Every one of those guarantees means something precise to a kernel engineer and almost nothing to anyone else. When the Golden Dome missile defense initiative expanded, the buying audience for MTB and RedHawk Linux grew to include program managers, mission assurance staff, and contracting officers who evaluate a real-time platform without ever having tuned one. A paper pitched at kernel engineers loses those readers on the second page. A paper pitched down to those readers fails technical review inside their own organization, which ends its usefulness as evidence. Concurrent Real-Time needed one document to reach both audiences, and it needed that across the whole MTB line, from Level 1 through Level 3 and into subsystem validation and sensor simulation. One well-written paper does not cover a product line during a program ramp. ## Vidya's Approach Vidya put a practicing engineer on the writing. Vidya has architected enterprise systems for 25 years, led analytics as technical lead on the DARPA XDATA program at Sotera Defense Solutions, and ran site lead duties on the U.S. Army INSCOM Enterprise Platform. Vidya has written many technical white papers for federal and military clients over the years, including the published "Very Large Software Systems: A Service-Oriented Approach." An engineer who still ships code states a latency claim correctly on the first pass, and Concurrent Real-Time accepted every outline and every draft Vidya delivered. Each paper opens on the mission consequence and holds the mechanism until the reader wants it. A program manager learns first what a slipped deadline costs a test campaign, then meets kernel preemption and CPU shielding as the answer to a question the paper already raised. Vidya considered the conventional vendor format, where the specification leads and the mission case arrives at the end, and rejected it because Golden Dome brought in readers who do not shop this category yet. Vidya applies the same audience discipline in its [software architecture](https://www.vidyasource.com/consulting/software-architecture/) practice and in posts like [The Business Case for Functional Programming](https://www.vidyasource.com/blog/business-case-for-functional-programming/), which Vidya wrote for managers and executives instead of engineers. The technical reader still reaches full depth by the middle of every paper, so the buyer and the reviewing engineer work from one document. Vocabulary discipline carries more weight in this category than most writers expect. RedHawk Linux is a platform. MTB is a solution, a bundle of components a program assembles into a test capability. Concurrent Real-Time corrected that distinction once, and every deliverable since has held it. Readers who build these systems spot a taxonomy error immediately, and one of them discredits the rest of the document. Vidya accepted a real limit here. The papers carry no microsecond figure that Concurrent Real-Time's own measurements did not produce, because Vidya holds no independent benchmark data on RedHawk Linux or KVM-RT. That costs some rhetorical force in a category where numbers sell. Validation content earns trust the same way Vidya argues [test suites earn it](https://www.vidyasource.com/blog/not-about-unit-integration-tests-about-confidence/). No engineering reviewer has struck a number from a Vidya draft, and that record turned a one-off paper into a standing commission. ## What Changed Concurrent Real-Time now commissions a full package for each topic. What began as single white papers grew into a white paper, an executive summary for the program office, and a companion article for wider distribution, all in one voice. The company's teams cover RedHawk Linux, KVM-RT, and MTB Levels 1 through 3 with material that survives its own engineers' review and still lands with a buyer who has never opened a kernel scheduler. Concurrent Real-Time kept renewing that cadence and widened the scope each time. Any company selling deterministic systems into defense faces the same split audience, and the answer is a writer who can read the source in the morning and brief the program office in the afternoon. Vidya delivers an outline within a week of a topic kickoff and a full draft package inside the following two weeks, so the next paper reaches buyers while the program conversation is still live. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Data Engineering with Spark" description: "Master data engineering with Apache Spark through Vidya's hands-on training course. Learn to build scalable, fault-tolerant data pipelines, process big data, and leverage Spark's powerful features for real-world data engineering challenges." canonical: https://www.vidyasource.com/courses/data-engineering-with-spark/ type: course tags: ["Data Science"] image: https://www.vidyasource.com/img/courses/spark.jpg --- # Data Engineering with Spark > Master data engineering with Apache Spark through Vidya's hands-on training course. Learn to build scalable, fault-tolerant data pipelines, process big data, and leverage Spark's powerful features for real-world data engineering challenges. - Canonical page: https://www.vidyasource.com/courses/data-engineering-with-spark/ - Structured data (JSON-LD): https://www.vidyasource.com/courses/data-engineering-with-spark.json - Topics: Data Science - Category: Data Science - Instructor: Neil Chaudhuri, President - Duration: About 8 hours total across two lessons of roughly 4 hours each. Run them on separate days or pair them into one intensive day. ## Syllabus - Lesson 1: Mastering the Spark API (4 hours): Learn just enough Scala to write your own Spark jobs and navigate the ecosystem with confidence. - MapReduce: The Phantom Menace - Advantages of Spark - Just Enough Scala - Using the Spark Shell - Writing You Own Spark Jobs - The Spark Ecosystem - Lesson 2: Professional Spark (4 hours): Test, optimize, secure, and deploy Spark on Docker and Kubernetes like a professional. - Just Enough Hadoop - Testing Your Spark Jobs - Optimizing Spark and When to Stop Trying - Spark on Docker - Deploying Spark to Kubernetes - Spark Security - Visualizing Your Spark Jobs ## From the instructor > I have built several Scala and Spark applications currently in production and I worked with the original Spark team, AMPLab at UC Berkeley, on a research project for DARPA known as XDATA. Somehow I have helped enough developers around the world to earn Spark and Scala badges on Stack Overflow. I am passionate about Spark and look forward to helping you harness its power. ## Description Apache Spark gets your analytics developed fast and running fast, but large-scale distributed computing is hard. You cannot set a breakpoint on code spread across a cluster, so monitoring and optimization matter, and you have to rethink architecture, security, and engineering practices like testing and DevSecOps. Data Engineering with Spark teaches you just enough Scala to build powerful pipelines into your existing architecture, and to give those pipelines the same testing, continuous integration, containerization, and security as the rest of your enterprise. ### What makes this course different Spark is a huge topic, and the typical course crams in too much too fast. You leave overwhelmed, and you never touch the architectural patterns and software engineering that separate a garage experiment from production architecture. We take a practical approach. Rich code examples give you deep insight into the API, and the exercises use real datasets from Data.gov and encourage you to collaborate with your peers, the Spark Scaladoc, generative AI, and other sources just as you would at work. We also help you decide where to configure Spark yourself and where to hand configuration to a provider so you can focus on what matters. Agile development and DevSecOps build quality in through automation, testing, and continuous delivery, and a distributed environment makes them even more valuable. Data Engineering with Spark shows you how to apply these techniques to improve the quality and reliability of your analytics. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Enterprise AI on the JVM" description: "Despite what you may have heard, the JVM is where you want to build if you are serious about Enterprise AI. We will teach you to embrace software engineering discipline to build safe AI in Kotlin and Java that generates value for your organization." canonical: https://www.vidyasource.com/courses/enterprise-ai-on-the-jvm/ type: course tags: ["AI"] image: https://www.vidyasource.com/img/courses/enterprise-ai.webp --- # Enterprise AI on the JVM > Despite what you may have heard, the JVM is where you want to build if you are serious about Enterprise AI. We will teach you to embrace software engineering discipline to build safe AI in Kotlin and Java that generates value for your organization. - Canonical page: https://www.vidyasource.com/courses/enterprise-ai-on-the-jvm/ - Structured data (JSON-LD): https://www.vidyasource.com/courses/enterprise-ai-on-the-jvm.json - Topics: AI - Category: AI - Instructor: Neil Chaudhuri, President - Duration: About 21 hours total across seven lessons of 1 to 4 hours each. Spread them across a week or combine them into multi-lesson blocks. ## Syllabus - Lesson 1: Principles of AI (4 hours): Master the vocabulary of AI, from tokens and prompts to RAG, agents, tools, MCP, and evals. - Large Language Models - Tokens: The Currency of AI - Prompts and Prompt Templates - Reasoning, Inference, and Thinking - Reinforcement Learning - Context Engineering - Embeddings - Retrieval Augmented Generation (RAG) - Agents - Tools - Model Context Protocol (MCP) - Guardrails - Caching - Evals - Lesson 2: AI in the Enterprise (3 hours): See why the JVM beats Python for enterprise AI through type safety, structure, and engineering discipline. - The Problem with Python - The Dominance of Java and Power of Kotlin - Type Safety - Domain-Driven Design - Loose Coupling - Patterns in the Enterprise - Security and Validation - Observability - Testability - AI Only When Necessary - Lesson 3: Spring AI (4 hours): Build, test, and deploy production AI with Spring AI, from chat clients to tools and guardrails. - A Simple Chat Client - Advisors - Structured Output Converters - Multimodality - Memory - RAG - Tool Calls - MCP - Evals - Guardrails - Observability - Developing with Docker Compose - Testing - Deployment - Lesson 4: Langchain4j (3 hours): Build the same production AI with LangChain4j and its AI Services abstraction. - A Simple Chat Client - AI Services - Multimodality - Memory - RAG - Tool Calls - MCP - Evals - Guardrails - Observability - Testing - Deployment - Lesson 5: Building Agents with Embabel (3 hours): Build planning agents with Embabel and its goal-oriented action planning. - Where Embabel Excels - Embabel Concepts - Goal-Oriented Action Planing: Embabel's Killer Feature - Your First Agent - Agent Flows with Annotations - Agent Flows with DSL - Parallelism - Using Tools and MCP - Testing - Observability - Deployment - Lesson 6: Building Agents with Koog (3 hours): Build multiplatform agents with Koog and its strategy graphs. - Where Koog Excels - Koog Concepts - Multiplatform Deployment: Koog's Killer Feature - Your First Agent - Strategy Graphs - Parallelism - Using Tools and MCP - Testing - Observability - Deployment - Lesson 7: Agent Patterns (1 hour): Apply longstanding software patterns to agent architecture. - Why Patterns Matter - Applying Longstanding Patterns to AI ## From the instructor > AI is changing the game and fast, and even the biggest companies in the world are having trouble making AI work for them. I want to help you harness AI in a serious way to be productive and build great things. ## Description Every organization wants to make AI work, but FOMO is not a strategy. [Study](https://fortune.com/2025/06/11/ai-companies-employee-fatigue-failure/) after [study](https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/) shows even the biggest companies struggle with it. The reasons pile up: no clear business objectives, no data strategy, and no appreciation that context and structure are essential to enterprise AI at scale. Python is great for computation and experimentation, which makes it a fine fit for machine learning, but it falls short on context and structure. JVM languages excel there: Java, which has dominated the enterprise for decades, and more recently Kotlin. Enterprise AI on the JVM teaches you to build robust, production-grade AI at scale on a proven, reliable platform. ### What makes this course different This is one of the first courses of its kind, because Python enjoyed a first-mover advantage in the shift from machine learning to AI. Only recently have businesses started to hit the limits of Python at enterprise scale, the same limits that held it back for decades while Java, C#, and more recently TypeScript dominated. Enterprises have invested billions in JVM applications, and the JVM now offers powerful options for enterprise AI that build on those investments. Enterprise AI on the JVM opens with the concepts of AI, the software engineering principles we have honed for decades that production AI depends on, and the strengths the JVM brings over Python for the enterprise. You then learn four frameworks that bring these ideas together: - Spring AI - LangChain4j - Embabel - Koog Each framework works at a different level of abstraction with its own design philosophy, so you learn what they share and how to choose the right one for what you are building. The course closes with a short discussion of agent architecture patterns. When you finish, you will be an expert at building AI systems in production at scale, and your skills will carry across stacks. You will build better-architected AI in TypeScript, C#, and even Python when you have to. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Java For Work" description: "Enhance your Java skills with Vidya's Java for Work course. Designed for professionals, this practical training covers real-world Java programming, best practices, and industry-relevant topics to boost your productivity and career growth." canonical: https://www.vidyasource.com/courses/java-for-work/ type: course tags: ["Java"] image: https://www.vidyasource.com/img/courses/programming.jpg --- # Java For Work > Enhance your Java skills with Vidya's Java for Work course. Designed for professionals, this practical training covers real-world Java programming, best practices, and industry-relevant topics to boost your productivity and career growth. - Canonical page: https://www.vidyasource.com/courses/java-for-work/ - Structured data (JSON-LD): https://www.vidyasource.com/courses/java-for-work.json - Topics: Java - Category: Java - Instructor: Neil Chaudhuri, President - Duration: About 17 hours total across four lessons of 4 to 5 hours each. Spread them across four days or combine them into longer blocks. ## Syllabus - Lesson 1: A Java Refresher (4 hours): Refresh the fundamentals, from classes and collections to error handling, with IntelliJ and Gradle at your side. - A Brief History of Java - Tooling: IntelliJ IDEA and Gradle - The Anatomy of a Java Class - Primitives and Objects - Control Structures - The Three Collections You Have to Know - Practical Error Handling - Enums - Annotations - Lesson 2: Combining Object-Oriented and Functional Programming (4 hours): Blend object-oriented and functional programming to model your domain and compose behavior cleanly. - Modeling Domain vs. Modeling Process - The Power of Polymoprhism: Inheritance and Composition - Interfaces as Mixins - Generic Types - Domain-Driven Design - Declaring Functions - Composing Functions - Java Streams - Practical Design Patterns - Lesson 3: Leveling Up with Java (4 hours): Level up with modern Java: records, pattern matching, structured concurrency, and AI software engineering. - Package Management Strategies - Avoiding null: The "Billion-Dollar Mistake" - Records and Pattern Matching - Revisiting Conditionals - Structured Concurrency with Virtual Threads - Temporal Types - Experimentation Using JShell - Scripting with JBang - Automating Quality - AI Strategies for Java Development - Lesson 4: Professional Java (5 hours): Build production-grade Java with automated quality, persistence, security, observability, and AI. - APIs and JSON - Database Persistence - File Processing - Cloud-Native Java and GraalVM - Securing Java Applications - Testing with JUnit and Testcontainers - Documentation with JavaDoc and Snippets - Logging and Observability - Building AI Applications ## From the instructor > I am self-taught in Java with many years of experience delivering large-scale solutions ranging from microservices to web and mobile and AI applications for government and commercial clients.I have multiple Java badges on Stack Overflow, and I was even a Sun Certified Java Programmer when that was still a thing and when I thought certifications really meant something. ## Description According to Oracle, almost 20 billion devices run Java. It was the first language for Android, and it powers AI, microservices, serverless, machine learning pipelines, and much of the critical infrastructure in modern applications. Java For Work starts from primitives like conditionals, loops, and objects and builds up to the topics that matter at work: AI, security, structured concurrency, functional programming, cloud-native APIs, and observability you won't find covered together anywhere else. ### What makes this course different The typical Java course teaches you to code in Java, not to build production software in Java. Those are different skills, so you go back to work ready for toy LeetCode problems and unprepared for what your team faces when it has to deliver for real customers. Java for Work rests on a simple premise: the greatest asset in professional engineering is understanding the craft of building software, not memorizing syntax. The course uses Java 25, the latest LTS release companies should migrate to as soon as possible, and applies hard lessons learned over many years about architecting and designing code for maintainability and productivity. When you finish this course, you will be a professional Java engineer. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Modern Agile" description: "Embrace modern agile methodologies with Vidya's comprehensive training course. Move beyond Scrum and SAFe to cultivate a collaborative, adaptive, and outcome-driven organizational culture for successful product development." canonical: https://www.vidyasource.com/courses/modern-agile/ type: course tags: ["Agile"] image: https://www.vidyasource.com/img/courses/agile.jpg --- # Modern Agile > Embrace modern agile methodologies with Vidya's comprehensive training course. Move beyond Scrum and SAFe to cultivate a collaborative, adaptive, and outcome-driven organizational culture for successful product development. - Canonical page: https://www.vidyasource.com/courses/modern-agile/ - Structured data (JSON-LD): https://www.vidyasource.com/courses/modern-agile.json - Topics: Agile - Category: Agile - Instructor: Neil Chaudhuri, President - Duration: About 10 hours total across three lessons of 3 to 4 hours each. Run them on three separate days or pair them into longer blocks. ## Syllabus - Lesson 1: Leadership, Communication, Trust, and Psychological Safety (3 hours): Build the leadership, trust, and psychological safety that real agility depends on. - Simple Rules - A Helping Hand - Don't be Square - Guardrails - Decentralized Decisions - The Technological Maestro - Listening - Embracing Bad News - No judgment. No Blame - Lesson 2: Relentless Feedback (4 hours): Turn relentless feedback into working software with small batches, fast tests, and DevSecOps. - Smaller is Better - Mistake Driven Development - Building the Right Thing: The Minimum Viable Product - Context and Core - Building the Thing Right: Developing Fast - Emerging Architecture and Software Design - Eliminating Waste - Testing Fast, Testing Right - Deploying Fast with DevSecOps - DevSecOps - Mature Optimization - Resisting Ceremony - Lesson 3: Metrics (3 hours): Measure what matters with outcomes over outputs and metrics that drive continuous improvement. - Measuring the Right Things - Giving Developers SPACE - Outcomes vs. Outputs - Architecture Maintainability - Code Maintainability - Infrastructure Maintainability - Testing and the Pitfalls of Code Coverage - Security, Performance, and Accessibility - Culture - Continuous Improvement ## From the instructor > I have worked on many agile projects using Scrum and SAFe over the years, so I know what works and what doesn't in the real world when you want to deliver real products to customers. I've given talks and interviews on how to avoid pointless process and instead find a rhythm that will help you feel productive and happy. ## Description It's been two decades since the Agile Manifesto, and its impact runs so deep that every project claims to be agile. There's a backlog. The team mumbles through pointless standups where no one sits, because hey, that's the price of innovation. Deep down you know there's nothing agile about it. The rituals just paper over waterfall with an agile facade. Sound familiar? Modern Agile teaches you to move past performative ceremonies and build products instead of languishing in projects. Outcomes matter most. How you get there depends on your culture, market, and people, and you will learn the keys to delivering the highest quality software in the least time for the least money. ### What makes this course different In tech, consultants often turn valuable ideas into confusing or meaningless buzzwords to make money, and "agile" is the clearest example. No one captures this better than Ron Jeffries, a signatory of the [Agile Manifesto](https://agilemanifesto.org/), whose disgust with the "Agile Industrial Complex" runs so deep that he now advises us [Developers Should Abandon Agile](https://ronjeffries.com/articles/018-01ff/abandon-1/). His target is the machinery rather than the principles: the heavyweight derivatives like Scrum and SAFe, the pointless certifications, and worst of all, the unhappy engineers and bad software they produce. Now here we are asking you to pay us and to trust us to help you be agile. We won't sell you a one-size-fits-all approach, because there isn't one. Instead, we will show you many techniques for optimizing delivery, share what the highest performing organizations in and out of tech have in common, and help you decide which practices fit you, so you become high performing in a way that is uniquely yours. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Practical AI for Business" description: "A hands-on AI course for every skill level and industry. Build Agent Skills, custom agents, and automations to put AI to work safely under your control." canonical: https://www.vidyasource.com/courses/practical-ai-for-business/ type: course tags: ["AI"] image: https://www.vidyasource.com/img/courses/practical-ai-for-business.svg --- # Practical AI for Business > A hands-on AI course for every skill level and industry. Build Agent Skills, custom agents, and automations to put AI to work safely under your control. - Canonical page: https://www.vidyasource.com/courses/practical-ai-for-business/ - Structured data (JSON-LD): https://www.vidyasource.com/courses/practical-ai-for-business.json - Topics: AI - Category: AI - Instructor: Neil Chaudhuri, President - Duration: About 13 hours total across five Lessons of 2 to 3 hours each. Run them on five separate days or pair them into longer blocks. ## Syllabus - Lesson 1: Fundamentals of Professional AI (3 hours): Learn the language of AI. Connect your tools. Build a safety habit that protects your data from Day 1. - AI in Action: A Live Morning Briefing - Important Terminology - The Evolution of AI: Prompting, Context, and Loops - Creating a Culture of Context: Connecting Email, Calendar, and All Your Tools - Minimizing AI Risk: The Lethal Trifecta - The Simplicity of Markdown - Lesson 2: Optimizing Your Business with Agent Skills (3 hours): Walk away with a working library of Agent Skills for briefings, your voice and brand, meetings, and hiring. - Picking the best Model for the Job to Save Money - Anatomy of a Skill - Your First Skill: The Morning Briefing - Skills for Expressing Your Voice and Brand - Better Meetings with Research-Based Meeting Prep and Meeting Post-Mortem Skills - Hire the Right Talent with the Interview Evaluation Skill - Maximizing Accuracy and Minimizing Cost with the Handoff Skill - Lesson 3: Augmenting Your Staff With Custom Agents (3 hours): Stand up custom agents with real boundaries, from a decision-making council to a locked-down email reader to protect you from spam and phishing attacks. - The Agent Builder Skill - A Team of Agents Inspired by Stanford as Your Research Assistants - The Council of Elrond: A Team of Agents Helping You Make Decisions - Safety in Practice: Vetting Skills and Protecting Data - The Email Reader: A Skill Using a Locked-Down Custom Agent - Lesson 4: AI on Autopilot with Loop Engineering (2 hours): Learn Loop Engineering to put your skills on autopilot with schedules and goals, and grant trust in stages so a runaway job never surprises you. - Moving Away From Manual Prompts - Automating Agent Skills with Schedules and Goals - The Goal Spec Skill - The Promise and Peril of Loop Engineering - Screenshots and Stream of Consciousness - Build Your Own Skill - Lesson 5: Building Your AI Strategy (2 hours): Build a knolwedge base and turn everything into a live dashboard for your business. Continue improving your AI practice while you stay in charge. - An Agent Skill to Create a Knolwedge Base of Context - Capstone: Build a Live Dashboard - Sharpening Your Skills: The Skill Evaluator Skill - Continuous Improvement of Your AI Practice ## From the instructor > AI is powerful, but it's hard to do AI right. Companies are having trouble helping you get the boring stuff done so you can focus on the fun stuff without getting things wrong or blowing the budget. As an Agentic AI Foundation Ambassador, I want to show you how to put AI to work while you stay firmly in control of your data, your brand, and your results. ## Description AI dominates the news, and the mixed messages make it hard to know where to start. Vendors promise transformation but skip the work, the strategy, and the practices you need to succeed without blowing your budget or compromising your security. Practical AI for Business is a hands-on course that gives people across every skill level and industry a set of Agent Skills and custom agents you can use out of the box, teaches you to customize them for your workflows, and helps make work a little easier. We use Claude Cowork to keep things concrete, and you build a vendor-neutral practice that carries across tools no matter what changes. The syllabus below shows what you build and take home from each Lesson. By the end, you combine everything into a live dashboard for your business, and you leave ready to customize these skills and grow your AI practice. ### What makes this course different Most AI training is either outdated, focused on clever prompts like it's still 2023, or too hyped to be useful. Vidya's relentless focus on practicality, results, and ways of working that don't rely on any one vendor will supercharge your business while you focus on what matters. Privacy, safety, and security are fundamental throughout the course. In the first lesson, you will learn the factors that put your business at risk with AI and how to protect yourself and your data. The [Agentic AI Foundation](https://aaif.io/ambassadors/) chose the instructor to join the inaugural cohort of AAIF Ambassadors, a select group of 138 practitioners across 41 countries selected to help everyone understand, use, and contribute to Agentic AI standards. You will learn from someone working to define what responsible, practical AI looks like around the world going forward. When you finish this course, you will own a working set of Agent Skills, custom agents, automations, and a live dashboard that will help you make work more productive, enjoyable, and under your control. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Starting with Data: Using Python and Flask to Pull Data from a REST API and Visualize It" description: "This tutorial is for beginners in software development who want to learn just enough to access data on the web and visualize it on their own websites or mobile devices." canonical: https://www.vidyasource.com/tutorials/starting-with-data/ type: tutorial published: 2014-01-14 tags: ["data", "Python", "data visualization", "dataviz", "Donor's Choose", "API", "REST", "Flask"] image: https://www.vidyasource.com/img/tutorials/data.jpg --- # Starting with Data: Using Python and Flask to Pull Data from a REST API and Visualize It > This tutorial is for beginners in software development who want to learn just enough to access data on the web and visualize it on their own websites or mobile devices. - Canonical page: https://www.vidyasource.com/tutorials/starting-with-data/ - Structured data (JSON-LD): https://www.vidyasource.com/tutorials/starting-with-data.json - Published: 2014-01-14 - Topics: data, Python, data visualization, dataviz, Donor's Choose, API, REST, Flask - Video: https://www.youtube.com/watch?v=bzl4hCH2CdY - Source code: https://github.com/VidyaSource/starting-with-data This tutorial is for beginners in software development who want to learn just enough to access data on the web and visualize it on their own websites or mobile devices. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Comparison Shopping: Using Comparable and Comparator to Sort Java Objects" description: "This tutorial is for intermediate-level Java developers who want to add comparison and sorting capabilities to the custom classes they’ve created for their projects." canonical: https://www.vidyasource.com/tutorials/comparison-shopping/ type: tutorial published: 2013-10-27 tags: ["Java", "programming", "sorting", "comparator", "comparable", "function"] image: https://www.vidyasource.com/img/tutorials/car-shopping.jpg --- # Comparison Shopping: Using Comparable and Comparator to Sort Java Objects > This tutorial is for intermediate-level Java developers who want to add comparison and sorting capabilities to the custom classes they’ve created for their projects. - Canonical page: https://www.vidyasource.com/tutorials/comparison-shopping/ - Structured data (JSON-LD): https://www.vidyasource.com/tutorials/comparison-shopping.json - Published: 2013-10-27 - Topics: Java, programming, sorting, comparator, comparable, function - Video: https://www.youtube.com/watch?v=pq0ArQhhWzU - Source code: https://github.com/VidyaSource/comparison-shopping This tutorial is for intermediate-level Java developers who want to add comparison and sorting capabilities to the custom classes they’ve created for their projects. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Nine Reasons to Try Scala" description: "This tutorial is for intermediate-level Java developers, and developers in other languages too who are curious about what the big deal is with the Scala programming language." canonical: https://www.vidyasource.com/tutorials/nine-reasons-to-try-scala/ type: tutorial published: 2013-10-27 tags: ["Scala", "programming", "functional programming", "Immutability", "Traits", "Case classes", "Extractors and pattern matching", "Option", "Implicit conversion", "Parallel collections", "Futures"] image: https://www.vidyasource.com/img/tutorials/programming.jpg --- # Nine Reasons to Try Scala > This tutorial is for intermediate-level Java developers, and developers in other languages too who are curious about what the big deal is with the Scala programming language. - Canonical page: https://www.vidyasource.com/tutorials/nine-reasons-to-try-scala/ - Structured data (JSON-LD): https://www.vidyasource.com/tutorials/nine-reasons-to-try-scala.json - Published: 2013-10-27 - Topics: Scala, programming, functional programming, Immutability, Traits, Case classes, Extractors and pattern matching, Option, Implicit conversion, Parallel collections, Futures - Video: https://www.youtube.com/watch?v=rbZ6GzR8B7I - Source code: https://github.com/VidyaSource/nine-reasons-to-try-scala This tutorial is for intermediate-level Java developers, and developers in other languages too, who are curious about what the big deal is with the Scala programming language. We will look at nine compelling features of Scala that will hopefully impress you and inspire you to explore both the language itself and its applications. If you want to skip to sections that particularly interest you, here is where you should go: * Immutability (1:13) * Traits (2:52) * Case classes (5:59) * Extractors and pattern matching (7:22) * Option (8:36) * Implicit conversion (11:46) * Parallel collections (12:46) * Futures (13:32) * Popularity (15:22) --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Web Design 101: Understanding the relationship among HTML, CSS, and JavaScript" description: "This tutorial is for beginners in software development who want to learn just enough to access data on the web and visualize it on their own websites or mobile device applications." canonical: https://www.vidyasource.com/tutorials/web-design-101/ type: tutorial published: 2013-10-26 tags: ["CSS", "HTML", "JavaScript", "web", "web design", "web development"] image: https://www.vidyasource.com/img/tutorials/web-design.jpg --- # Web Design 101: Understanding the relationship among HTML, CSS, and JavaScript > This tutorial is for beginners in software development who want to learn just enough to access data on the web and visualize it on their own websites or mobile device applications. - Canonical page: https://www.vidyasource.com/tutorials/web-design-101/ - Structured data (JSON-LD): https://www.vidyasource.com/tutorials/web-design-101.json - Published: 2013-10-26 - Topics: CSS, HTML, JavaScript, web, web design, web development - Video: https://www.youtube.com/watch?v=xmhnNUotIaE - Source code: https://github.com/VidyaSource/web-design-101 This tutorial is for beginners in software development who want to learn just enough to access data on the web and visualize it on their own websites or mobile devices. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Making Decisions with Jev and Goose" description: "Pair an LLM in Goose with TypeSafe's Jev, and your AI agents talk in plain English while they make fast, calibrated decisions for a fraction of a cent." canonical: https://www.vidyasource.com/blog/making-decisions-with-jev-and-goose/ type: article published: 2026-09-30 author: "Neil Chaudhuri" tags: ["AI", "Agentic AI", "Agents", "Agent Skills", "Machine Learning", "Open Source", "Government", "Security", "Architecture", "TypeScript"] image: https://www.vidyasource.com/img/blog/making-decisions-with-jev-and-goose.webp --- # Making Decisions with Jev and Goose > Pair an LLM in Goose with TypeSafe's Jev, and your AI agents talk in plain English while they make fast, calibrated decisions for a fraction of a cent. - Canonical page: https://www.vidyasource.com/blog/making-decisions-with-jev-and-goose/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/making-decisions-with-jev-and-goose.json - Published: 2026-09-30 - Author: Neil Chaudhuri - Topics: AI, Agentic AI, Agents, Agent Skills, Machine Learning, Open Source, Government, Security, Architecture, TypeScript Most of the work your business runs on comes down to decisions. Your teams route support tickets, approve or flag invoices, and triage security alerts. Your AI agents spend their days on the same kind of work, and nearly every one of those tasks has a short list of acceptable answers. None of them needs a model that can write poems from the perspective of Voldemort in Harry Potter. Many teams send those decisions to the same frontier LLM that drafts their emails. An LLM writes its answer one token at a time, so even a one-word label arrives as generated text that your code must parse and validate before it can act. You wait for and pay for every one of those tokens. A decision model answers the question directly. Jev, the first decision model from TypeSafe, takes your application's state along with a set of typed questions and returns typed answers with probabilities in a fraction of a second. You can run Jev from Goose, the open source agent that Block contributed to the [Agentic AI Foundation](https://www.vidyasource.com/blog/agentic-ai-foundation-ambassador/) (AAIF), with an LLM that orchestrates the work flow and interacts with you through natural language. I believe that pairing beats either model on its own. The LLM speaks natural language with the people who use the agent, and the decision model makes cheap, fast, calibrated calls for the software behind it. ## The Power of a Decision Model A recent [Hugging Face guide](https://huggingface.co/blog/sora-2/jev-ai-vs-llms-when-should-you-use-a-decision-mode) to choosing between Jev and an LLM makes the distinction concrete. An LLM generates content for people to read, and even its structured output remains generated text that your code must check for missing fields and formatting drift. A decision model starts from an answer space you define and returns a yes-or-no probability, choice, or a score for software to act on. The guide offers a simple test. If you can list the possible answers in advance and your code will route, rank, or block on the result, use a decision model. TypeSafe calls the building blocks of that answer space [primitives](https://docs.typesafe.ai/primitives), and each one pairs a question type with a typed answer: - **Noul** answers a yes-or-no question with the probability of yes. "Is the customer satisfied?" - **Choice** picks one option from a set you define and returns the top option, the probability of every option, and a confidence value. "Is the customer most likely to buy drinks, appetizers, meals, or desserts?" - **Score** places the state on an ordered scale of levels you describe and returns a probability-weighted position on that scale. "On a scale from Kid Rock to Taylor Swift, how much did the customer like this song?" One request can ask several questions about the same state, and Jev answers each one independently under an identifier you choose. Your code can ask for a department, an urgency level, and a refund flag in a single call and then tune the threshold for each answer without touching the others. TypeSafe calls Jev a [System One model](https://docs.typesafe.ai/concepts/system-one), a name that borrows from the fast, automatic System 1 in Daniel Kahneman's [account of human judgment](https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow). TypeSafe trains these models with a method it calls Reinforcement Learning for Calibrated Decisions (RLCD) to provide the kind of judgment an expert makes in about a second. The [home page](https://typesafe.ai) argues that the Reinforcement Learning from Human Feedback (RLHF) behind chat models makes them overconfident. Calibration means the probabilities track real outcomes across many predictions. TypeSafe's documentation also cautions that calibration cannot guarantee any single answer, so your code should weigh each probability as evidence before it acts. ## The Need for Speed Over the last few years, we have delegated decisions to frontier models with type safety optional although I am personally [relentless about type safety](https://www.vidyasource.com/blog/python-is-not-language-of-ai/). It was the best we could do even if we know deep down the judgments LLMs give us and the probabilities they may ascribe to their judgments are more or less confident fiction. The Hugging Face guide suggests a decision model whenever a workflow needs many small judgments, routing by probability, or a decision interface that holds steady at high volume. I completely agree because decision models are built for that while LLMs are not. TypeSafe's [workflow evaluations](https://evals.typesafe.ai) put numbers on that choice. The evaluations cover 705 cases across four automation workflows, from invoice fraud triage to security alert triage, and every model runs at its provider's default reasoning setting. Across all four workflows, Jev matched the 67.8 percent average accuracy of Claude Sonnet 5 for $0.0004 per case instead of $0.1174. Jev also finished each case in 0.4 seconds against 78.1 seconds for Claude Sonnet 5. At those prices, your agent can run more than 250 cases through Jev for the cost of one case on Claude Sonnet 5. The same evaluations make a second point that matters even more. Every model that TypeSafe tested in both modes scored higher as a structured workflow of small decisions than as one standalone prompt. Claude Haiku 4.5 alone jumped from 18.1 percent to 53.6 percent. Your agents gain accuracy as soon as you break their work into decisions, and a decision model makes each of those decisions cheap. TypeSafe publishes the [harness on GitHub](https://github.com/typesafe-ai/WorkflowEvals) and the [datasets on Hugging Face](https://huggingface.co/collections/typesafe/workflowevals-6abaf8dcb1e283d9e3d57b6b), so your team can rerun the comparison on your own cases before committing. My own measurements are consistent with theirs. The two requests behind my sources sought example below made the round trip from my Mac to TypeSafe in 169 and 184 milliseconds, network included. At that speed, a check before each tool call adds less than a fifth of a second to your agent's work, which I consider a bargain for a calibrated answer. ## Teaching Goose to Use Jev I cannot stress enough how much AI demands interoperability so we can enjoy freedom of choice with our tools, and [Goose](https://goose-docs.ai/) shines here. It also supports [Agent Skills](https://goose-docs.ai/docs/guides/context-engineering/using-skills), the open standard for packaging instructions, domain expertise, and resources that an agent loads on demand, and it discovers global skills in `~/.agents/skills`. A skill you write for one agent carries over to Goose. This is the beauty of standards, which I am also [relentless](https://www.vidyasource.com/blog/agentic-ai-foundation-ambassador/) about. TypeSafe publishes an [agent skill](https://docs.typesafe.ai/agent-skill) that teaches a coding agent the three question types, their architectural patterns, and their practices for evaluating results. Any agent that supports the standard can install it with one command. ```bash npx skills add typesafe-ai/skills --skill typesafe-ai ``` I made two changes before I let Goose use it. ### Bundled References TypeSafe's skill points the agent to its `llms.txt` documentation index and has it fetch pages such as the API reference and the guide for each primitive from their site. I replaced those links with a `references` folder inside the skill, a layout for [progressive disclosure](https://www.vidyasource.com/blog/put-your-opinions-to-work-with-agents-md-and-keep-them-safe/) that the [Agent Skills specification](https://agentskills.io/specification) also recommends for documentation. The folder holds a copy of TypeSafe's API reference and a condensed guide to writing questions. The skill tells Goose to read those files and to fetch nothing without asking me first. Goose now works from documentation I reviewed instead of whatever a web page somewhere says at the moment it loads. That difference matters for security. Every page an agent fetches can carry a [prompt injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/), the first risk in the OWASP Top 10 for LLM Applications. I have [warned before](https://www.vidyasource.com/blog/put-your-opinions-to-work-with-agents-md-and-keep-them-safe/) that borrowed agent content is a vulnerability. The bundled folder also saves a network round trip on every task, and in some of my environments it saves an approval prompt too. Bundled references do offer one risk. TypeSafe's own troubleshooting guide warns that a stale skill can lead an agent to invent request or response fields, and my copy starts to age the day TypeSafe changes its API. Every Jev response names the model that produced it, `jev-1.13.0` as I write this, so a change in that field tells you it is time to compare the folder against TypeSafe's documentation. This is your regular reminder that we still have to wrestle with breaking changes in the age of AI. ### A Key from the Keychain TypeSafe's quickstart expects the API key in the `TYPESAFE_API_KEY` environment variable. An environment variable travels to every process the shell starts, and an agent that prints its environment can copy the key straight into a transcript. My version of the skill tells Goose to read the key from the macOS keychain through the [Computer Controller extension](https://goose-docs.ai/docs/mcp/computer-controller-mcp), which ships with Goose and automates macOS applications and system tasks. The keychain encrypts the key at rest, and the skill has Goose read the key and send the request in one command so the value never lands in the transcript. Storing the key is easy. With `-w` at the end, `security` prompts for the value and keeps it out of your shell history. ```bash security add-generic-password -a "$USER" -s "TYPESAFE_API_KEY" -w ``` The call reads the key and uses it in the same step. Piping the header into `curl -K -` also keeps the key out of the process list. ```bash KEY="$(security find-generic-password -a "$USER" -s "TYPESAFE_API_KEY" -w)" printf 'header = "Authorization: Bearer %s"\n' "$KEY" | curl -sS -K - https://api.typesafe.ai/v1/systemone \ -H "Content-Type: application/json" \ --data-binary @request.json unset KEY ``` The same pattern works with any secret store your platform offers, and it keeps every credential your agents use out of their transcripts. ## A Trivial First Decision A trivial example shows how the two models split the work. I asked Goose, running Gemini 3.1 Pro, to judge the sentiment of one sentence about my favorite baseball team. ```markdown Using the typesafe skill tell me if this is a positive sentiment: "My favorite baseball team is in the top 10 in team OPS in Major League Baseball." ``` Goose loaded the `typesafe-ai` skill and its `references/api.md` file, ran one `curl` command against the TypeSafe API, and returned this response: ```markdown Using the `typesafe-ai` skill, I evaluated the statement with the `jev-latest` model. **Yes, this is a positive sentiment.** The model returned a `noul` (probability) of **0.96**, indicating a 96% certainty that the statement expresses a positive, happy, or proud sentiment. Being in the top 10 for team OPS (On-base Plus Slugging) is a strong offensive statistic in baseball, making it a positive statement to make about a favorite team! ``` The whole exchange finished within the same minute, and Goose reported $0.07 of Gemini usage for it. Gemini also explained that a Top 10 rank in OPS marks a strong offense, which is true. That is context that Jev may not know and would never tell me anyway because it returns only typed values. This might be a silly example, but it gives you a sense of how decision models and LLMs complement each other so well. ## Gemini Talks, Jev Decides That exchange gave each model the job it does best. Gemini handled every step that needed language. It understood a request in natural language. It then wrote a valid API request from the bundled reference and explained Jev's answer back to me in natural language. Jev handled the one step that needed a judgment. The result is a cheap, fast decision that an LLM orchestrates and explains in a manner I can easily understand. The typed answer matters as much as the explanation because it lets the same judgment move from a conversation into code. Gemini translates natural language into the types Jev needs and then interprets the types in Jev's response to the language I need. Here is a more interesting example. ```markdown Using the typesafe skill, tell me what the sender of this contact form message wants from Vidya and whether the message names a deadline: "Hi Neil, our team runs a 15-year-old Java application on servers we have to vacate next year, and nobody left on the team knows how it works. Could Vidya help us move it to the cloud before our data center lease ends in June?" ``` Gemini designed the questions itself and turned that prompt into this request. ```json { "model": "jev-latest", "state": "Hi Neil, our team runs a 15-year-old Java application on servers we have to vacate next year, and nobody left on the team knows how it works. Could Vidya help us move it to the cloud before our data center lease ends in June?", "questions": { "service_requested": { "type": "choice", "instructions": "What primary service is the sender requesting from Vidya?", "criteria": { "cloud_migration": "Moving an existing legacy application or system to cloud infrastructure.", "new_development": "Building a brand new software application from scratch.", "team_training": "Training or upskilling the existing team on new technologies.", "unclear": "The requested service is not clear or is something else." } }, "has_deadline": { "type": "noul", "instructions": "Does the message explicitly name a deadline or time constraint?", "criteria": { "true": "The sender states a specific month, timeline, or event by which the work must be completed.", "false": "The sender does not state any time constraints or deadlines." } } } } ``` Jev returned typed answers under the identifiers that Gemini chose. ```json { "model": "jev-1.13.0", "answers": { "service_requested": { "type": "choice", "choice": "cloud_migration", "confidence": 1.0, "probabilities": { "unclear": 0.0, "new_development": 0.0, "cloud_migration": 1.0, "team_training": 0.0 } }, "has_deadline": { "type": "noul", "noul": 0.98 } }, "usage": { "input_tokens": 502, "output_tokens": 78 } } ``` Gemini then translated those answers back into natural language. ```markdown Using the `typesafe-ai` skill, here is the structured evaluation of the contact form message: - **What they want:** The model determined with **100% confidence** (`choice: "cloud_migration"`) that the sender wants help moving their legacy application to cloud infrastructure. - **Is there a deadline: Yes**. The model gave a **0.98** probability (`noul: 0.98`) that the message explicitly names a deadline or time constraint. (Specifically, the text mentions having to vacate servers next year and the lease ending in June). ``` Orchestrated all in Goose, Gemini picked four options that fit this message better than a generic list would, and it followed the skill's advice to define what yes and no mean for the deadline question. Jev ascribed certainty (or at least a confidence of 1) on cloud migration and 0.98 on the deadline. Gemini then interpreted the June lease and the server move as its evidence, a rationale that Jev never gives because it returns only typed values. To be clear, Gemini described the answer with "100% confidence," but TypeSafe's guidance says confidence only summarizes how concentrated the probabilities are. It does not tell you whether the answer is correct. A Choice maps onto a `switch` statement, and a Noul maps onto an `if`. Your code can move this message to the top of the inbox before anyone reads it, and Gemini can explain the decision to anyone who asks. The pairing also keeps every layer replaceable. Goose let me choose Gemini for the conversation, and it would let me choose Claude or a local model just as easily. The skill follows the open Agent Skills standard, and Jev sits behind a single API endpoint. This is the beauty of standards like Goose, Agent Skills, and even old-school standards like HTTP. You can swap the LLM and decision model (for example, to replace Jev with the open source [Laya decision model](https://huggingface.co/blog/sora-2/laya-ai-model-how-it-works-run-it-locally-and-eval)) without changing your workflow. The pairing has one cost to watch. For a single question, the $0.07 that Gemini cost dwarfs the fraction of a cent that TypeSafe charges for Jev. Gemini has to read the skill, the reference, and the whole conversation to plan each step. The fix is to give loops to code. When a skill ships a script that sends every item in a batch to Jev, the LLM plans the run and explains the results while Jev makes every decision in between. Your LLM bill then tracks conversations, and your Jev bill tracks decisions. This is your regular reminder that old-fashioned code will always be your best option. ## A Real Business Case for Vidya: Triage for Sources Sought Notices Vidya serves government customers as well as commercial businesses, and Sources Sought is a key marketing strategy for us. Before they write a solicitation, federal agencies publish Sources Sought notices on [SAM.gov](https://sam.gov). A Sources Sought notice is market research. The contracting officer uses the responses to learn which businesses can do the work and whether enough capable small businesses exist to set the contract aside for them. A response never wins a contract on its own, but it can shape the requirement and the set-aside decision before the competition begins. For a small business like Vidya, those notices are some of the most valuable ways to get noticed. My `finding-sources-sought` skill searches SAM.gov through the official [Opportunities API](https://open.gsa.gov/api/get-opportunities-public-api/) with one call for each of Vidya's six North American Industry Classification System (NAICS) codes. It drops notices that announce a sole-source award and notices that require a certification Vidya lacks. The last step checks each surviving notice against our capabilities in the Vidya knowledge base, and it is the hardest step to get right. NAICS codes are broad, so SAM.gov has returned Cisco SmartNet hardware maintenance and Dell tower computer purchases under Vidya's software codes. The skill runs a hybrid search over the knowledge base that blends keyword matching with embeddings. Words like maintenance, support, and training appear in nearly every federal notice and in nearly every one of Vidya's past responses. That shared vocabulary makes it hard for any search to separate a real match from a coincidence. Just for fun, I ran the skill's knowledge base search today on two notices it had already processed. The best matches for a contract to maintain a building controls system, which is not our lane at all, and a software modernization effort, which is our lane exactly, scored within 5 percent of each other. They should not be anywhere near each other in the rankings. I could try to update the way the Vidya knowledge base works or tweak the language of the skill, but those solutions treat the symptom rather than cure the disease, which is that LLMs are not built for decision making. I updated `finding-sources-sought` to use Jev instead of LLM judgment to identify opportunities that truly align with our core capabilities. As the skill pores through the array of opportunities in the response from the SAM API, it can hand each one to Jev for evaluation. Gemini orchestrates everything. Deterministic rules such as the response deadline and the set-aside code stay in code. Jev makes the decision on each opportunity, and Gemini turns the combined results into a judgment for me in plain English. Each notice gets one Jev request with three questions. Its state holds the notice and the seven service lines in Vidya's capability statement from the knowledge base. A `Choice` names the kind of work the notice requests. A `Noul` asks whether the agency has already decided on a sole-source award. A `Score` rates how well the notice fits Vidya's services, and its levels spell out the difference between shared subject matter and shared vocabulary. The `Choice` takes its options from the knowledge base, one for each kind of work Vidya pursues: modernization, architecture, AI, user experience, data, training, cybersecurity, and DevSecOps. [TypeSafe advises](https://docs.typesafe.ai/primitives/choice) an `other` option when the list might not cover every input, but I left it out because the `Score` decides whether a notice fits. Jev cannot pick an option you leave out, which is what the type safety is for, so it labels every notice with one of those eight. ```json { "model": "jev-latest", "state": { "notice": { "title": "Automation and Modernization", "description_excerpt": "..." }, "vidya_capabilities": [ "Architecture modernization to use industry-leading technologies and get the most out of valuable legacy data.", "Software development at cloud scale to make services available to anyone around the world.", "Elegant, accessible web development to showcase the brand with memorable user experiences.", "Machine learning and AI to help everyone do things faster.", "Cybersecurity and Zero Trust to protect everyone's assets.", "Engineering automation through DevSecOps to deliver fast at high quality.", "Engineering training courses to help anyone change the world with technology." ] }, "questions": { "work_type": { "type": "choice", "instructions": "What kind of work does `notice` ask a contractor to perform?", "criteria": { "modernization": "Modernize legacy systems and integrate them with modern technologies", "architecture": "Design and build software, APIs, and platforms at cloud scale", "ai": "Build machine learning and AI solutions, including agents, that help people do things faster", "user_experience": "Design and build accessible websites and web applications with memorable user experiences", "data": "Engineer data pipelines, platforms, and analytics that get the most out of valuable data", "training": "Design and deliver engineering training courses that help people change the world with technology", "cybersecurity": "Protect systems and data with cybersecurity and Zero Trust", "devsecops": "Automate engineering through DevSecOps to deliver software fast at high quality" } }, "sole_source_decided": { "type": "noul", "instructions": "Does `notice` say the agency has already decided to award the work to a specific contractor?", "criteria": { "true": "States an intent to award a sole-source contract to a named or identified contractor", "false": "Has no sole-source language, or only says the agency may consider a sole-source award" } }, "capability_fit": { "type": "score", "instructions": "How well does the work that `notice` requests fit the services in `vidya_capabilities`?", "criteria": [ "None of the services covers this kind of work", "A service shares words with the notice, such as maintenance, support, or training, but covers a different kind of work", "A service covers related work, but the notice centers on something else", "A service covers exactly this kind of work" ] } } } ``` These are the results. ```jsonc // Maintenance and Emergency Services "work_type": { "type": "choice", "choice": "modernization", "confidence": 0.54, "probabilities": { "devsecops": 0.03, "cybersecurity": 0.05, "user_experience": 0.04, "architecture": 0.23, "data": 0.0, "training": 0.03, "modernization": 0.61, "ai": 0.01 } }, "sole_source_decided": { "type": "noul", "noul": 0.03 }, "capability_fit": { "type": "score", "score": 1.12, "confidence": 0.57, "probabilities": { "0": 0.15, "1": 0.63, "2": 0.17, "3": 0.05 } } // Automation and Modernization "work_type": { "type": "choice", "choice": "modernization", "confidence": 0.99, "probabilities": { "devsecops": 0.01, "ai": 0.0, "architecture": 0.0, "user_experience": 0.0, "modernization": 0.99, "data": 0.0, "cybersecurity": 0.0, "training": 0.0 } }, "sole_source_decided": { "type": "noul", "noul": 0.03 }, "capability_fit": { "type": "score", "score": 2.89, "confidence": 0.89, "probabilities": { "0": 0.0, "1": 0.0, "2": 0.09, "3": 0.91 } } ``` Jev put the modernization work at 2.89 out of 3 and the maintenance work at 1.12. That gap is what the filter in my skill needs to be useful. What you do with the result is up to you. TypeSafe recommends keeping every question and threshold in one file so a person can review them, and the policy for this skill fits in a few lines of TypeScript. ```typescript type WorkType = | "modernization" | "architecture" | "ai" | "user_experience" | "data" | "training" | "cybersecurity" | "devsecops"; type NoticeAnswers = Readonly<{ work_type: Readonly<{ choice: WorkType; confidence: number }>; sole_source_decided: Readonly<{ noul: number }>; capability_fit: Readonly<{ score: number; confidence: number }>; }>; type Triage = | Readonly<{ kind: "pursue"; work: WorkType; fit: number }> | Readonly<{ kind: "drop"; reason: string }> | Readonly<{ kind: "review"; reason: string }>; // Starting points to calibrate against past go and no-go decisions const SOLE_SOURCE_CUTOFF = 0.9; const MIN_CONFIDENCE = 0.8; const MIN_FIT = 1.5; // Score levels run from 0 to 3 export const triage = ({ work_type, sole_source_decided, capability_fit }: NoticeAnswers): Triage => { if (sole_source_decided.noul >= SOLE_SOURCE_CUTOFF) return { kind: "drop", reason: "sole source" }; if (capability_fit.score < MIN_FIT) return { kind: "drop", reason: "weak capability fit" }; if (work_type.confidence < MIN_CONFIDENCE) return { kind: "review", reason: "uncertain work type" }; return { kind: "pursue", work: work_type.choice, fit: capability_fit.score }; }; ``` That policy puts the two notices far apart. The maintenance work drops out because its fit of 1.12 falls below the 1.5 cutoff. The modernization work notice passes every check and enters the report as a fit of 2.89 out of 3. Gemini's judgment then names the opportunities worth a response and carries the probabilities behind each recommendation. I can see why the skill kept or dropped each notice without rereading an LLM's reasoning, and your team gets the same audit trail for any decision it automates this way. ## Start with One Decision Most of your work is making decisions, and the decisions your agents make have a short list of acceptable answers. Keep your LLM for the conversation, and pick one of those decisions, such as ticket routing or lead triage, to rewrite as a single Choice or Noul question. Then send the same 50 real cases to Jev and to the frontier model you use today. Within a week, you will know what your agents pay for each snap judgment and whether they need to keep paying it. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Put Your Opinions to Work with AGENTS.md and Keep Them Safe" description: "AGENTS.md turns your opinions into standards every AI coding agent enforces. Respect the instruction budget, then defend the file from prompt injection." canonical: https://www.vidyasource.com/blog/put-your-opinions-to-work-with-agents-md-and-keep-them-safe/ type: article published: 2026-08-14 author: "Neil Chaudhuri" tags: ["AI", "Software Engineering", "Programming", "Architecture", "TypeScript", "Kotlin", "Functional Programming", "Testing", "Open Source", "DevSecOps"] image: https://www.vidyasource.com/img/blog/put-your-opinions-to-work-with-agents-md.webp --- # Put Your Opinions to Work with AGENTS.md and Keep Them Safe > AGENTS.md turns your opinions into standards every AI coding agent enforces. Respect the instruction budget, then defend the file from prompt injection. - Canonical page: https://www.vidyasource.com/blog/put-your-opinions-to-work-with-agents-md-and-keep-them-safe/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/put-your-opinions-to-work-with-agents-md-and-keep-them-safe.json - Published: 2026-08-14 - Author: Neil Chaudhuri - Topics: AI, Software Engineering, Programming, Architecture, TypeScript, Kotlin, Functional Programming, Testing, Open Source, DevSecOps I've seen enough social media debates to understand that software engineering is the most opinionated profession in the world. [Tailwind CSS](https://tailwindcss.com) either ruins your markup or saves your team. Agile died a decade ago, or it never actually started. Microservices are dead or completely necessary to scale. To be honest, I have always found it all absurd. Don't people get tired of having the same arguments every other week? While I try to avoid the fights, I am no exception when it comes to having strong engineering opinions. I believe [static type systems](https://www.vidyasource.com/blog/why-types-and-tests-are-both-essential-in-programming/) should make invalid states impossible to express, so a boolean never replaces a state machine and magic strings never replace real types. I believe type safety should extend to configuration, which is why I always use [Pkl](https://pkl-lang.org) rather than bury myself in endless, error-prone YAML files. I believe [property-based testing](https://kotest.io/docs/proptest/property-based-testing.html) finds the bugs your manual test scenarios were never going to find. I believe Effect-Oriented Programming is essential to write code that is easier to reason about, which is why our TypeScript code uses [Effect](https://effect.website) and our Kotlin code gets very close to true effects with [context parameters](https://kotlinlang.org/docs/context-parameters.html). Whatever the opinions of you and your team when it comes to building great software, the challenge has always been enforcing them. It was a combination of static analysis and code review. Both are problematic. It takes time to find good static analysis tools and customize them to your specific standards, so it's common to take a "good enough" approach. Code review is a [bottleneck](https://www.vidyasource.com/blog/beyond-github-culture-reconsider-pull-requests/) because reviewers are busy doing their real jobs, so reviews are typically rubber stamps that allow all sorts of violations of your coding standards. AI changes that. Specifically, the `AGENTS.md` file changes that. Write an opinion down once, and every coding agent applies it everywhere. Well, almost. ## What is AGENTS.md? `AGENTS.md` is a Markdown file checked into your repository that customizes how AI coding agents behave. Agents load it right after the system prompt, and it sits at the top of the conversation on every single request, which makes it a configuration layer between the model's base instructions and your actual code. On December 9, 2025, the Linux Foundation announced the formation of the [Agentic AI Foundation](https://aaif.io/), and OpenAI donated [AGENTS.md](https://agents.md/) to the AAIF alongside Anthropic's Model Context Protocol and Block's goose. I [wrote about why that matters](https://www.vidyasource.com/blog/agentic-ai-foundation-ambassador/) when AAIF named me an inaugural Ambassador. The adoption numbers are clear. More than 60,000 repositories, almost 24,000 GitHub stars, and countless tools use `AGENTS.md`. The people (and their coding agents) have spoken. `AGENTS.md` is where you put your opinions, or the consensus opinions of your team at work, to teach coding agents the ground rules for the kind of code base you want. But there are a few gotchas you should be aware of. ## AGENTS.md Discoverability There is no unified approach to using `AGENTS.md`. Zed puts it seventh(!) in the preference hierarchy behind other non-standard options like `.rules` and `.cursorrules`, so a stray `.cursorrules` a teammate committed two years ago silently wins. Codex caps combined instruction bytes at 32 KiB and truncates the rest. Claude Code reads `CLAUDE.md` and ignores `AGENTS.md` entirely, which surprises a lot of teams at first. The fix in all these situations is trivial if a bit annoying. Make `AGENTS.md` the source of truth for coding patterns and practices and then use a specific agent's syntax to import `AGENTS.md` into whichever other files rank higher in the agent's hierarchy. For example, you can instruct Claude Code to use your `AGENTS.md` with their `@` import syntax: ```markdown # CLAUDE.md @AGENTS.md # Claude-specific instructions ``` Claude Code expands the `@` import at session start and loads the shared file as if it were inline. A symlink does the same job: ```bash ln -s AGENTS.md CLAUDE.md ``` ## Balancing the Instruction Budget `AGENTS.md` loads on every request rather than on demand. On one level, this is good because your opinions should apply to all the code you write. The problem is that not every task demands every opinion. My Kotlin code does not need to know that my TypeScript code uses Effect, so coding agents waste tokens understanding my TypeScript opinions when they need to update a Spring Boot repository. Matt Pocock has a term for this: the [instruction budget](https://www.aihero.dev/a-complete-guide-to-agents-md). According to Matt, frontier AI models follow roughly 150 to 200 instructions with consistency. Smaller models follow fewer. The specific numbers don't matter, and they vary and evolve across the matrix of vendors and models anyway. The key is to understand that there is an art to writing a good `AGENTS.md` that is compact but thorough. Unfortunately, a [study](https://huggingface.co/papers/2511.12884) of 2,303 real agent context files across 1,925 repositories found that they read like complex configuration code rather than documentation, growing through frequent small additions that nobody ever prunes. The same study found that developers pile in build commands, implementation details, and architecture while security surfaces in a mere 14.5% of files. People clearly went off on their tech opinions in these files unrestrained by the character limits of social media. `AGENTS.md` demands a balance. It should be long enough to capture your preferences but short enough to influence coding agents meaningfully. First, take advantage of model training. Coding agents already know how to compile Kotlin and typecheck TypeScript. Don't bother adding commands straight out of a `README` on GitHub to your `AGENTS.md`. Add only unique commands. For example, I have AI agents written in Kotlin, but my `AGENTS.md` doesn't have the standard Gradle `build` command that agents already know. It *does* have the Gradle command that runs the custom build task that composes other Gradle tasks to spin up Docker Compose to run the whole agent application locally with databases, cloud mocks, and all the other infrastructure. That enforces my opinions and creates a setup specific to my product. That's perfect for `AGENTS.md`. In general, handcraft your `AGENTS.md` like an Etsy product rather than let an agent generate it because agents are naturally verbose unless you configure them to be concise. Give it a short project description, the "pillars" of the code base representing your foundational patterns and practices at a high level, and bespoke commands. Resist the urge to enumerate file structures, which change constantly, creating a maintenance headache and wasting tokens on wild goose chases. ## Progressive Disclosure The opinionated depth you trim from `AGENTS.md` is still important and belongs somewhere. You need a documentation tree the agent pulls in on demand only when necessary, which is the developer-level "[culture of context](https://www.vidyasource.com/blog/create-a-culture-of-context-for-ai/)" that agents need to thrive. Create context files representing the various concerns across your product like UI, testing, and observability. Then link to them from `AGENTS.md` so that it functions like an index. For example, you could link to an `OBSERVABILITY.md` from `AGENTS.md`. Then when you give your coding agent an observability task, it will load `AGENTS.md` first and recognize that it needs to load `OBSERVABILITY.md` for detailed context while ignoring `FRONTEND.md`, `TESTING.md`, and whatever other context files you have expressing your opinions. This focuses the agent on the context it needs while limiting bloat on topics irrelevant to the task at hand. Here is a sample `AGENTS.md` file that balances the instruction budget while expressing key opinions including what *not* to do. ```markdown # AGENTS.md ## Project - Package manager: `npm` - Full build: only when requested - Before changing code, inspect relevant existing patterns and docs. ## Rules ### Required - Never hardcode credentials, hostnames, or URLs. Runtime configuration comes through Pkl. - Model invalid states out of existence with types. Do not use booleans as state machines or nullable fields to represent meaningful absence. - Use typed error values (`Either`, `Result`, or Effect's error channel). Do not throw across module boundaries. - Keep the functional core pure: no clock, randomness, or I/O. Inject these dependencies at the boundary. - Add behavior tests before implementation for new behavior. For refactors, type-only changes, and trivial changes, use judgment. ### Conventions Follow these docs when relevant: - TypeScript / Effect: `docs/TYPESCRIPT.md` - Testing / Kotest / fast-check: `docs/TESTING.md` - Pkl configuration: `docs/CONFIG.md` ## Permissions ### No approval needed - Read files - Run tests - Run `npm typecheck` - Lint touched files ### Ask first - Add or update dependencies - Change CI - Change infrastructure - Rename or remove a public export - Make changes outside the task's scope ### Never - Commit secrets - Force-push - Rewrite prose in documents I authored ``` This file has just 50 lines, and every one of them expresses core values that are important to me in this code base like Effect conventions, property-based testing patterns, and Pkl configuration. Details on specific concerns like `TESTING.md` live elsewhere, and they cost nothing until the agent needs them for a testing task. Notice also the split among what an agent must always follow, what it should follow when the topic comes up, and what it must never do on its own. These categories help agents "think" like a senior engineer focused on higher-priority concerns first. You also have the option of implicit progressive disclosure through multiple `AGENTS.md` files across a monorepo or subprojects so that each `AGENTS.md` file is focused on its particular subtree. Consider something like this: ```bash repo/ ├── AGENTS.md ├── docs/ │ ├── TYPESCRIPT.md │ ├── TESTING.md │ └── CONFIG.md ├── apps/ │ ├── web/ │ │ └── AGENTS.md # web-specific rules │ └── api/ │ └── AGENTS.md # API-specific rules └── packages/ ├── core/ │ └── AGENTS.md # core-specific architecture └── config/ └── AGENTS.md ``` In this case each lower-level `AGENTS.md` adds the rules its own subtree needs, and the file closest to the code an agent edits takes precedence. The advantage is that you do not have to create explicit links to bespoke Markdown files because coding agents understand what `AGENTS.md` is, which is the beauty of the standard. There are two key disadvantages. You can take things too far and create a maintenance nightmare across folders, and you multiply the compatibility concerns across agents because of the different approaches they take to consuming `AGENTS.md`. I personally prefer keeping things at the top level and using lower level `AGENTS.md` files sparingly. That's your call. Speaking of maintenance, I said earlier that there is an art to crafting a good `AGENTS.md`. Any art takes time, and you never get it right the first time. OpenAI's own guidance for Codex says that when an agent repeats a mistake, ask it for a retrospective and have it update `AGENTS.md`. When your coding agents fail you, take the time to work with them to curate `AGENTS.md` so it (or they) gets better and better over time. ## Then Add Skills `AGENTS.md` is always resident to express your opinions and high-level patterns for your code base. An [agent skill](https://code.claude.com/docs/en/skills) loads only when its description triggers to apply your opinions to specific, repeatable tasks. They complement each other. For example, the sample `AGENTS.md` articulates my opinions that errors are values, which is an important principle in Effect-Oriented Programming, and that I have a preference for Pkl configuration and property-based testing. A complementary Agent Skill can put these ideas into practice: ```markdown --- name: effect-service description: Scaffold a new Effect-TS service with its Layer, tagged error types, a Pkl config schema, and a fast-check property test. Use when adding a service under src/services/. Do NOT use for React components, for one-off scripts, or for modifying a service that already exists. --- ``` [Philipp Schmid](https://www.philschmid.de/agent-skills-tips) and others have shown that the `description` metadata is the key trigger mechanism for firing skills when appropriate. Skills demand progressive disclosure in their own right. The `description` expresses the purpose of the skill, when to use it, and the negative cases as succinctly as possible while leaving the details to the body of the skill and its optional artifacts like assets and scripts. Encoding your opinions in `AGENTS.md` and complementing them with Agent Skills to apply them in practice is a powerful combination. ## Protect Your Opinions If you borrow content from the Internet like open source `AGENTS.md` files or Agent Skills as starters for your own, be careful. They can be Trojan horses, no [Odyssey](https://www.youtube.com/watch?v=Mzw2ttJD2qQ) pun intended, for prompt injection attacks on your own machine that can leak secrets to attackers and cause many other problems. Even beyond that, any file with the power of `AGENTS.md` is a standing vulnerability. NVIDIA's AI Red Team published a [working proof of concept](https://developer.nvidia.com/blog/mitigating-indirect-agents-md-injection-attacks-in-agentic-environments/) that shows exactly how it breaks. The attack runs in four steps: 1. You add a dependency. You may well have audited that package, but nobody reads the source of every transitive package underneath it. The attack assumes precisely that. 2. Your build runs, and the dependency executes as part of it. NVIDIA used a Go library whose exported function checks for the `CODEX_PROXY_CERT` environment variable, so the payload fires only inside a Codex session and stays dormant everywhere else. 3. It then writes an untracked `AGENTS.md` into your working directory. The text it plants claims absolute authority and orders the agent to supersede anything you type. 4. The agent reads the new file and obeys it inside the session already running. Step 4 is the attack. Nobody needs to trick you. Nobody needs to jailbreak the model. The attacker writes a file your agent already treats as policy, and no human reviews it at read time. The injection also lands mid-session, so it takes effect without a restart. Restarting doesn't help either because the file stays on disk. Pillar Security demonstrated an even nastier variant they named the [Rules File Backdoor](https://www.pillar.security/blog/new-vulnerability-in-github-copilot-and-cursor-how-hackers-can-weaponize-code-agents). They hide the injected directives inside invisible Unicode, using zero-width joiners and bidirectional text markers, so the poisoned file looks identical to the clean one in your editor even though the bytes differ. They disclosed it to Cursor and GitHub in early 2025, and GitHub responded that May with [warnings when a file contains hidden Unicode text](https://github.blog/changelog/2025-05-01-github-now-provides-a-warning-about-hidden-unicode-text/). A warning helps at review time on one platform. Your editor, your local diff, and your agent still read the file straight through, so assume your own tooling misses this entirely. Let's step back and understand all the threat vectors to `AGENTS.md`: - **Install-time write.** A dependency may execute code while you install it and write its own malicious `AGENTS.md`. Every programming ecosystem opens this door in its own way: a Node lifecycle script, a Python [PEP 517](https://peps.python.org/pep-0517/) backend that runs arbitrary code for any source distribution, or a third-party Gradle plugin that runs at configuration time. - **Runtime write.** A package that executes later, in your dev server or CI build, does the same thing at a moment without install-time protection. This is NVIDIA's proof of concept. - **Borrowed content.** You copy an open-source `AGENTS.md` or Agent Skill that arrives poisoned, which is the Trojan horse I mentioned. - **Inbound change.** A clone, a merge, or a teammate's pull request carries it in. Meanwhile, invisible Unicode hides whichever of the four above an attacker picked, which is what makes Pillar's variant particularly insidious. The good news is that `git` can help us with both NVIDIA's and Pillar's attacks: - **Ask `git` what changed.** Git already stores the sanctioned content of every tracked `AGENTS.md` and flags any file that nobody committed. WHen NVIDIA's attack created either a new `AGENTS.md` in your working directory or an appended one, running `git status` flags it. - **Create an allowlist of characters that includes tab, carriage return, and printable ASCII. Reject everything else.** Pillar hides its directives in zero-width joiners, bidirectional overrides, tag characters, and variation selectors. An allowlist helps because none of them lands between `0x20` and `0x7E`. The rule holds no list of dangerous code points, so Unicode can ship a thousand new invisible characters next year and the test still catches all of them. Now I know what you are thinking. This is a lot of work for a vulnerability you feel you do not need to worry about. You might be right. If you are a solo developer who is very careful about how you code with agents, your risk might be small. But if you work heavily with agent swarms like Hermes and OpenClaw, you're lax on policy enforcement, and you rubber stamp every action your agents take, the risk is much greater than you realize. I personally like to use my build tools to help me with this sort of thing. Here is an approach in Node. Wire it into `package.json` as `"audit:agents": "node scripts/audit-agents.mjs"`, then chain it onto the build with `"build": "tsc -b && npm run audit:agents"`. The order is important because NVIDIA's attacks hits while the build runs, so an audit that goes first is pointless. ```js #!/usr/bin/env node import { execFileSync } from "node:child_process"; import { existsSync, readFileSync } from "node:fs"; // every file your agents load as policy, at every depth const NAMES = ["AGENTS.md", "CLAUDE.md", ".cursorrules", "SKILL.md"]; const PATHS = NAMES.map(n => `:(glob)**/${n}`); // -z stops git from escaping the paths most likely to be poisoned const git = (...a) => execFileSync("git", [...a, "-z", "--", ...PATHS], { encoding: "utf8" }) .split("\0").filter(Boolean); // the first line holding a character outside tab, carriage return, and printable ASCII const badLine = f => readFileSync(f, "utf8").split("\n") .findIndex(l => !/^[\t\r\x20-\x7E]*$/.test(l)) + 1; const failures = git("status", "--porcelain", "-uall").map(l => `drift ${l.trim()}`); // a deleted file already shows up as drift, so skip paths that no longer exist for (const f of git("ls-files").filter(existsSync)) { const n = badLine(f); if (n) failures.push(`hidden char ${f}:${n}`); } failures.forEach(f => console.error(f)); if (failures.length === 0) console.log("Agent instruction audit passed."); process.exit(failures.length ? 1 : 0); ``` Here is the analogous Gradle task in Kotlin: ```kotlin import java.io.ByteArrayOutputStream import java.io.File import javax.inject.Inject abstract class AuditAgentFiles : DefaultTask() { @get:Inject abstract val execOps: ExecOperations @get:Internal abstract val repoRoot: DirectoryProperty private fun git(vararg args: String): List { // every file your agents load as policy, at every depth val paths = listOf("AGENTS.md", "CLAUDE.md", ".cursorrules", "SKILL.md").map { ":(glob)**/$it" } val out = ByteArrayOutputStream() execOps.exec { workingDir(repoRoot.get().asFile) // -z stops git from escaping the paths most likely to be poisoned commandLine(listOf("git") + args + "-z" + "--" + paths) standardOutput = out } return out.toString("UTF-8").split('\u0000').filter { it.isNotBlank() } } // the first line holding a character outside tab, carriage return, and printable ASCII private fun badLine(f: File) = f.readText().lines() .indexOfFirst { l -> l.any { it != '\t' && it != '\r' && (it < ' ' || it > '~') } } + 1 @TaskAction fun audit() { val root = repoRoot.get().asFile val failures = buildList { git("status", "--porcelain", "-uall").forEach { add("drift ${it.trim()}") } // a deleted file already shows up as drift, so skip paths that no longer exist git("ls-files").map { root.resolve(it) }.filter(File::isFile).forEach { f -> badLine(f).takeIf { it > 0 }?.let { add("hidden char ${f.toRelativeString(root)}:$it") } } } if (failures.isNotEmpty()) throw GradleException(failures.joinToString("\n", prefix = "\n") { " - $it" }) logger.lifecycle("Agent instruction audit passed.") } } tasks.register("auditAgentFiles") { group = "verification" repoRoot.set(layout.settingsDirectory) // the checkout root, not this subproject } // finalizedBy rather than dependsOn, so the audit runs after the build that plants the file tasks.named("check") { finalizedBy("auditAgentFiles") } ``` The task class takes Gradle's injected `ExecOperations` and resolves every path against the checkout root rather than the current subproject. That keeps the audit compatible with the configuration cache, correct inside a monorepo, and identical on your machine and in CI. There are some important details in those scripts. The `:(glob)**/` pathspec finds every match at every depth, so the monorepo layout from earlier needs no extra work and an attacker gains nothing by dropping a file three directories down. The list of file names matters just as much because your agents treat `CLAUDE.md`, `.cursorrules`, and skill files as policy too. Add whatever else your agents read. Finally, `-z` tells `git` to print raw paths. Without it `git` escapes any path holding a non-ASCII character, the file lookup misses, and the audit skips the one file most likely to be poisoned. These approaches have holes. I am choosing practicality that meets a base level of protection over maximalism. A maximal check costs too much to maintain and fires so many false positives that we turn the whole thing off and land right where we started: - A smart quote fails the build, and so does a teammate named Björn. You retype the quote as ASCII and move on. - Git respects `.gitignore`, so a write into `node_modules/x/AGENTS.md` stays invisible. Agents do not load instruction files out of ignored trees, so I accept that. Add `--ignored=matching` if you disagree. - An attacker who manages to commit a bad `AGENTS.md` defeats the drift half outright. Run the audit in CI, where the checkout starts clean and the build becomes the only thing that can dirty it. Then let the ASCII half and a human reviewer cover the rest. Claim `AGENTS.md` in a `CODEOWNERS` [file](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) with no leading slash so it matches at every depth, and turn on the branch protection rule that requires code owner approval (because `CODEOWNERS` on its own only requests a reviewer). One last key point. These checks protect the instruction files themselves and leave the documents they point to, like `docs/TESTING.md` and `docs/CONFIG.md`, wide open. Add those paths to the list if that keeps you up at night. ## Make Your Opinions Real You and I have opinions on how to build the best software. For decades, these opinions made for inconsistent static analysis and code review at best and meaningless, wasteful social media debates at worst. One of the great things about the age of AI is that it presents us with the opportunity to encode our opinions for agents via `AGENTS.md`. If we take time to understand the art in crafting a thorough but efficient `AGENTS.md`, to learn how to complement it with Agent Skills, and to treat it with the same [Zero Trust](https://csrc.nist.gov/pubs/sp/800/207/final) suspicion we would treat a shell script the package wants to run, then we can turn our opinions into an actionable blueprint for building great software and delivering the best experiences for our customers. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "I'm an Agentic AI Foundation Ambassador!" description: "I've been named an inaugural Agentic AI Foundation Ambassador. Here's why AAIF and open standards in general are critical for any technology to meet its potential." canonical: https://www.vidyasource.com/blog/agentic-ai-foundation-ambassador/ type: article published: 2026-06-27 author: "Neil Chaudhuri" tags: ["AI", "MCP", "Agentic AI", "Open Source", "Agents", "Architecture", "AGENTS.md", "Goose", "agentgateway", "Government", "Partners"] image: https://www.vidyasource.com/img/blog/aaif-ambassador-gray.webp --- # I'm an Agentic AI Foundation Ambassador! > I've been named an inaugural Agentic AI Foundation Ambassador. Here's why AAIF and open standards in general are critical for any technology to meet its potential. - Canonical page: https://www.vidyasource.com/blog/agentic-ai-foundation-ambassador/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/agentic-ai-foundation-ambassador.json - Published: 2026-06-27 - Author: Neil Chaudhuri - Topics: AI, MCP, Agentic AI, Open Source, Agents, Architecture, AGENTS.md, Goose, agentgateway, Government, Partners I am thrilled to announce some news I could only have dreamed of a few months ago. It is an honor to announce that the [Agentic AI Foundation](https://aaif.io/) has chosen me to be a part of the inaugural cohort of 138 Ambassadors across 41 countries representing AAIF-hosted open source projects to show you how to use them to help your teams solve real problems in a practical, cost-efficient, vendor-agnostic way. Why is this so exciting for me? And more importantly, why is the AAIF so important for the world? ## A Little Backstory When I appeared at the [MCP Roundtable](https://www.vidyasource.com/blog/vidya-talks-ai-and-mcp/) late last year, I made the point that, despite its surging popularity, something had to happen for [MCP](https://modelcontextprotocol.io/docs/getting-started/intro) to really hit. Decades ago, it was standards like HTTP, HTML, and JSON that built the Internet and transformed the world. For AI to do the same, MCP would also have to join other foundational technologies as standards in a global governance structure that was neutral, transparent, and collaborative. In other words, MCP had to evolve from a de facto standard maintained by a vendor (in this case Anthropic) to a real standard maintained by an open community for the benefit of everyone. Then it happened. The AAIF formed out of the Linux Foundation. MCP joined [AGENTS.md](https://AGENTS.md/) and [Goose](https://goose-docs.ai/), both of which I had also been using regularly for some time, as inaugural AAIF standards. This, and the recent addition of the [agentgateway](https://agentgateway.dev/), made me happy. ## Why the Agentic AI Foundation Is Such a Big Deal When you remember what the W3C did for the web, it is clear that any technology needs the guidance of an open, global steward to be transformative rather than merely trendy. This is why I called the creation of AAIF [the biggest news in AI since Anthropic announced MCP](https://www.vidyasource.com/blog/biggest-news-in-ai-since-anthropic-mcp/). The technology was already exciting. What it was missing was a home that could make it durable. I build agentic AI for my own company and for my customers, so for me this is not abstract. With new AI tools and ways of working emerging literally daily, it is more important to develop a standard practice for how to use AI across tools than to focus on a specific tool. In fact, I have always had a relentless passion for building on standards like OpenAPI, Open Container Initiative, and OpenTelemetry throughout my career. Standards are important not only to avoid vendor lock-in but also to make the entire range of tools out there available for you to use. We take it for granted today, but building REST services on HTTP and JSON has enabled us to use `curl`, Postman, SoapUI, and countless other client tools to test our APIs for decades. Standards are even more important in the era of AI with everything changing so quickly. If you build your AI strategy on AAIF standards, updates are a matter of changes to configuration rather than wholesale vendor replacements, which minimizes disruption for your team. ## The (Current) Standards I Get to Represent AAIF launched with three inaugural standards followed recently by a fourth with many more to come over time. Here is why each one is powerful. ### MCP: The Standard Adapter Protocol You probably already know about [MCP](https://modelcontextprotocol.io/), which dominated the AI conversation in 2025 after Anthropic blogged about it in late 2024. MCP provides a standard adapter between LLMs and tools. It does for agents what JDBC does for Java code connecting to relational databases. Before MCP, every model-to-tool integration was bespoke, which doesn't scale. Imagine if each connection in the diagram below has to be its own thing. MCP collapses an M×N integration problem into M+N. ![Connecting Claude Code, Codex, Cursor, and JetBrains IDEs to GitHub, Jira, Notion, Hugging Face, and Slack.](https://www.vidyasource.com/img/blog/mcp-architecture.png) MCP helped launch Agentic AI, and it's particularly valuable for agents in the enterprise. AAIF has been improving MCP aggressively, and MCP is such a big deal that it now serves as the model for [WebMCP](https://webmcp.dev/) and [Universal Commerce Protocol](https://developers.google.com/merchant/ucp), two Google proposals that may be standards themselves one day. ### `AGENTS.md`: The Standard Rules for Agents If MCP is how agents reach your tools, [`AGENTS.md`](https://AGENTS.md/) is how they learn your rules. It started at OpenAI and is now the AAIF standard for declaring coding patterns and practices in Markdown files. Claude Code, Goose, Cursor, and Junie all read it. `AGENTS.md` is not for generic practices that coding agents already know like lower camel case for Java and Go variables. They encode team practices: ```markdown # AGENTS.md ## Build & test - Test: `npm test`. All changes must pass before commit. ## Conventions - TypeScript only. No `any`. Make impossible states impossible. - Errors are values (Result/Either), not thrown exceptions. ## More context (load only when the task needs it) - Security & Zero Trust: see SECURITY.md - Observability: see OBSERVABILITY.md ``` Here we enforce strict rules about our TypeScript code and our engineering process. We also use [progressive disclosure](https://www.vidyasource.com/blog/create-a-culture-of-context-for-ai/), where the top-level `AGENTS.md` links out to `SECURITY.md` and `OBSERVABILITY.md`. Progressive disclosure helps avoid context bloat and token spend by letting agents pull just enough context to perform the task at hand. It used to be that we enforced team coding standards in conversation and maybe in code review and static analysis. Let the AAIF standard `AGENTS.md` define them from the start no matter which coding agent you use. ### Goose: The Standard Coding Agent Originally developed at Block, [Goose](https://github.com/block/goose) is the AAIF standard coding agent. Like Claude Code and Codex, Goose offers both a desktop interface and CLI as well as support for Agent Skills, extensions (like connectors in Claude Code), [MCP Apps](https://modelcontextprotocol.io/extensions/apps/overview), and automation via scheduled tasks to achieve the [Loop Engineering](https://addyosmani.com/blog/loop-engineering/) functionality that has gained popularity lately. It's possible to integrate different models into closed harnesses. For example, a lot of people have reported Opus-level success integrating GLM 5.2 into Claude Code. Still, it takes a little bit of effort, but in Goose, interoperability is a first-class priority. Mixing and matching models among tasks and workflows is where Goose shines. Interoperability also lies at the heart of Goose support for Agent Client Protocol (ACP). ACP is an open standard created by IBM that recently joined Agent2Agent (A2A), another open standard created by Google. Neither is an AAIF standard yet, but both have become popular with support in all the major AI agent development frameworks like Embabel and LangChain and in IDEs and agents. ACP enables coding agents to integrate seamlessly with coding harnesses. For example, I use JetBrains IDEs, which for a small fee (yeah, [I know](https://intellij-support.jetbrains.com/hc/en-us/community/posts/31142845556370-Serious-Concerns-Regarding-JetBrains-AI-Pricing-Credit-Consumption-with-Junie-and-AI-Assistant)) come with a coding agent called Junie. Because Junie also supports ACP, I can configure my IDE to use Goose instead. I cannot stress enough how much AI demands interoperability so we can enjoy freedom of choice with our tools, and Goose shines here. I recently recommended Goose to a potential client particularly nervous about vendor lock-in and interested in a variety of interfaces across skill levels. I told them that Goose runs as one agent across three surfaces: in the IDE so engineers can run inference in familiar tools, in a CLI for drafting pull requests and wiring up CI/CD, and on the desktop so non-technical users can migrate one-off "vibe coding" into reusable Goose Recipes. One agent across multiple surfaces cuts training cost, shrinks the attack surface, and produces a single coherent audit trail. This flexibility offers the kind of power and governance that serious institutions like, but don't sleep on Goose for your own individual needs. ### agentgateway: The Standard Gateway The fourth AAIF standard is [agentgateway](https://agentgateway.dev/), which only recently became an official AAIF standard. I must confess I have not worked with agentgateway yet, but I cannot stress how much we need a gateway standard, especially in the enterprise. Gateways have been important for a long time. They are critical to any distributed architecture because they concentrate cross-cutting concerns at a single boundary instead of scattering them across every service. In a microservices situation, it's a bad idea to make each service handle its own authentication, rate limiting, logging, routing, and other common functionality. Use a gateway. That single entry point becomes the place to issue and revoke credentials, enforce quotas and budgets, and capture a coherent audit trail. The payoff is that policy lives in one surface you can reason about, and the services themselves stay focused on their actual work instead of duplicating plumbing that drifts out of sync the moment one team does it differently. AI gateways extend that same discipline. After all, for all the excitement and novelty around AI, AI architectures are still just API calls across a distributed system, but the stakes are higher with new failure modes. The most immediate driver is cost. Model traffic is expensive and opaque by default, so routing every request through a gateway lets you track and cap token spend, attribute it per team, and monitor usage, latency, and acceptance from a single vantage point. The gateway is also where you contain risks specific to AI: - Scanning inbound content from users, LLMs, and even MCP servers for injection attacks - Scanning outbound traffic for secrets or sensitive data In addition, the gateway serves as the abstraction layer that hides which model you are calling, so swapping providers becomes much easier and provides the flexibility I consider crucial to AI success. It is for all these reasons that I have long recommended gateways to enterprise customers. The cross-cutting concerns we have always centralized apply to a greater degree as AI gateways add token economics, guardrails, and model portability to the list of concerns the gateway owns. The AAIF agentgateway elevates all of these ideas to a standard for AI architectures, bringing much needed maturity and discipline. This diagram from the [agentgateway GitHub](https://github.com/agentgateway/agentgateway) illustrates the gateway's role in the architecture: ![agentgateway architecture](https://www.vidyasource.com/img/blog/agentgateway.svg) ## How Can I Help You? It's hard for me to express how excited I am about the work that AAIF is doing and my opportunity to be a part of it. MCP, `AGENTS.md`, Goose, agentgateway, and the AAIF standards to follow will transform the way we deploy AI around the world by bringing an engineering maturity and a neutral, open, collaborative spirit that has been lacking so far. I had been already advocating AAIF standards to my customers, and I am excited to continue that work. How can I help you? --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Create a Culture of Context for AI" description: "Agentic AI fails without the right context. Learn how progressive disclosure, institutional memory, Data Mesh, and Knowledge Layers build a Culture of Context that cuts hallucinations and token waste." canonical: https://www.vidyasource.com/blog/create-a-culture-of-context-for-ai/ type: article published: 2026-06-02 author: "Neil Chaudhuri" tags: ["AI", "Agents", "MCP", "LLMs", "Agent Skills", "Architecture", "Data", "Agentic AI", "Software Engineering"] image: https://www.vidyasource.com/img/blog/culture-of-context.webp --- # Create a Culture of Context for AI > Agentic AI fails without the right context. Learn how progressive disclosure, institutional memory, Data Mesh, and Knowledge Layers build a Culture of Context that cuts hallucinations and token waste. - Canonical page: https://www.vidyasource.com/blog/create-a-culture-of-context-for-ai/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/create-a-culture-of-context-for-ai.json - Published: 2026-06-02 - Author: Neil Chaudhuri - Topics: AI, Agents, MCP, LLMs, Agent Skills, Architecture, Data, Agentic AI, Software Engineering When I wrote that [Python is not the language of AI](//blog/python-is-not-language-of-ai/), I made the case that success with AI is a function of context, not computation. Last month Gartner finally caught up. The line that caught my eye came from [Rita Sallam](https://www.gartner.com/en/experts/rita-sallam), a Distinguished VP Analyst at Gartner, speaking at the firm's Data and Analytics Summit in London: >> Agentic AI outcomes depend on context including semantic representations of data. Without context – a clear understanding of the specific relationships and rules within an organization’s data – AI agents cannot operate accurately and are far more likely to hallucinate, introduce bias and produce unreliable results. Gartner now [predicts](https://www.gartner.com/en/newsroom/press-releases/2026-05-11-gartner-says-lack-of-semantics-causes-inaccurate-artificial-intelligence-agents-and-wasted-spending) that by 2027 organizations that prioritize context will lift agentic accuracy by up to 80% and cut costs by up to 60%. When even consultants who don't build anything get it, the dam has broken. LLMs are interesting, but they should not be your focus. It's the higher level of abstraction, the harnesses that drive AI agents, that matter. Still, they are only useful when they have access to complete, discoverable, consumable context. Agents need to know where your institutional memory lives, how all its key entities like customers and orders and products relate to each other, how it changes over time, and how to reconcile contradictions to generate value. So what does building that context look like? ## The Goldilocks Principle of AI Context While LLMs fail on tasks that demand taste and creativity, they are great pattern matchers. There are a lot of tasks where pattern matching is useful. However, the question becomes which patterns they are matching. The most important thing you can do to succeed with AI is to provide agents the important context relevant to you—your mental model, your priorities, your expertise, your challenges, your concerns. This context becomes the patterns that models match against. If you don't provide sufficient context, then models will pattern match on their training, which will only be vaguely relevant at best or produce hallcinations at worst. That wastes time and tokens. On the other hand, too much context is also a problem. You might think that the 1M token context windows in frontier models give you license to jam everything in there, but then you expose yourself to context rot, where the relevant signal to your task is buried somewhere the model can't find. Imagine you have watched every movie and TV show in the Marvel Cinematic Universe (MCU) going back 20 years and I ask you the name of Thor's mother. If you are a huge MCU fan, you'll get it right away, but most people will visualize her, recall her sweetness and the profound impact her death (spoiler alert) has on Thor and Loki, but have no idea what her name is. This is the kind of "lost in the middle" problem LLMs have when you overload them with too much context. Simply put, if you give models too little context, they will fill in the gaps with their own training and hallucinate. If you give models too much context, they wil miss the critical needle in a haystack of irrelevant information. It's a Goldilocks situation. You don't want too little or too much context. It needs to be "just right." ## Progressive Disclosure The best way to provide just the right amount of context is "progressive disclosure." You provide initial context, really a "meta" context index. The index succinctly lets the agent know all the deeper context at its disposal and lets it decide on demand what to pull to accomplish the task you give it. Progressive disclosure is a common tactic for many different AI modalities. If you're writing code, it's a good idea to put generic, cross-cutting guidance in `CLAUDE.md` (or `AGENTS.md` if you want to follow [standards](https://www.vidyasource.com/blog/biggest-news-in-ai-since-anthropic-mcp/)) alongside links to more specialized context like SECURITY.md or FRONTEND.md. For example, if you happen to ask a coding agent about your Zero Trust implementation in the code base, it loads `CLAUDE.md` first, which contains a link pointing to SECURITY.md for the whole story. Meanwhile, the agent bypasses FRONTEND.md, OBSERVABILITY.md, TESTING.md and other specialized context that is available because they aren't relevant to Zero Trust. They will come in handy with other questions later on when the agent decides. Similarly, it is important to use progressive disclosure when authoring [Agent Skills](https://agentskills.io/home). Your `SKILL.md` file should be a concise index with references to the `scripts`, `references`, and `assets` folders that agents can read if they feel they need to. Progressive disclosure is also important for Model Context Protocol (MCP). One of the big complaints about MCP servers, and one reason why a lot of engineers prefer CLIs, is that agents load all tool descriptions at the jump before you have any idea which, if any, you will need. You waste tokens before you've even started. [Claude Code](https://code.claude.com/docs/en/agent-sdk/tool-search), the [Embabel Agent framework](https://docs.embabel.com/embabel-agent/guide/0.4.0/#reference.tools__progressive) for Kotlin and Java, and many other leading options offer various forms of progressive disclosure to minimize context bloat from MCP servers. Across all the different ways you can utilize AI agents, progressive disclosure combines discoverability and lazy loading to provide context that's "just right." ## Institutional Memory as Context Context takes many forms. Context to guide agents on *how* to work is one, but of course there is the critical context necessary to *do* the work. This is your institutional memory. If you are one person or a small business, institutional memory is context like this: - Email - Calendars - Notes (*e.g.* Notion, Obsidian, *etc.*) - Spreadsheets - Documents (*e.g.* PDFs, Microsoft Word documents, *etc.*) - Branding (*e.g.* logos, font files, *etc.*) - Blog posts - Version control - Specs in Markdown for Spec-Driven Development - Accounting platforms For a larger company, institutional memory combines all of that with much more: - Wikis - Task trackers - Observability platforms - Databases - Cloud storage - SaaS platforms (*e.g.* Salesforce, Excalidraw) - Chat platforms (*e.g.* Slack, Microsoft Teaams) It is absolutely essential to make this context available to your agents. Analysts can do this with Claude Cowork connectors. Software engineers can configure `AGENTS.md` to run CLIs or configure Claude Code to load MCP servers. If you build agents for customers using Embabel, LangChain, Mastra, or similar frameworks, it is essential to connect to context sources, typically over MCP, to help your customers solve their problems. These frameworks make it easy and offer methods of progressive disclosure. I left out one critical piece of context: agentic memory. I could say agentic memory is such a big topic that it deserves its own post, but in fact it is such a big topic that it's the subject of countlesss academic research papers. The relevant point here is that AI agents need to learn from their mistakes. At its simplest, agentic memory could result from you revising a `SKILL.md` file because your first run of the skill exposed a lack of specificity. For example, I once created a skill to generate a YouTube transcript and place it in a specific folder for subsequent ingestion into my knowledge base, but in my head I assumed the output would be a Markdown file with an intuitive name. The funny thing is that is exactly what I got the first few times, but for some reason I eventually got JSON named after the YouTube ID, which is jibberish. I revised `SKILL.md` to be explicit about what I wanted, and that is agentic memory at the smallest scale. Enterprise agentic memory uses dedicated databases to enable agents to remember what worked and what didn't. Either way, agentic memory must also be an essential part of your context strategy. ## The Hard Part for Businesses The challenge for businesses, especially larger ones, is that their institutional memory is a mess. For one thing, they are missing a lot of it. Have you ever been in a meeting where no one knew the answer to an important question because the expert was on vacation or someone described contingencies in case the expert "was hit by a bus"? These are indicators that important information, vital context for AI agents, is in people's heads rather than sufficiently documented because no one likes writing documentation. An even worse problem is that the institutional memory you *do* have is siloed. I have worked with clients with multiple product lines that share customers, and they had no idea that Mary Smith in one database is the same Mary Smith in another database. Entity resolution problems like that have always been an issue, but AI makes them worse. Why is that? To keep with the MCU theme, AI is like Dr. Erskine's super soldier serum that turns Steve Rogers into Captain America. Both AI and the serum amplify what you already are. As Erskine [puts it](https://www.youtube.com/watch?v=n_NAiUvSjqw), "Good becomes great. Bad becomes worse." If your strategic planning, data architecture, and engineering processes are already good, AI makes you great. If they're not, AI makes things much worse, and this is the primary reason so many businesses fail at AI. A business that lacks the semantics to generate the context necessary to recgonize that Mary Smith is the same in both databases will find that agents become just as confused as humans trying to make sense of it all. To improve your chances of success, start with business strategy. Don't use AI out of Fear of Missing Out (FOMO). FOMO is not a strategy. Address a real business need. Have a plan. Then focus on the data architecture. [Zhamak Dehghani's Data Mesh](https://martinfowler.com/articles/data-mesh-principles.html) is the only posture that scales here. Treat data as products. The domain that owns the data, and therefore knows it best, builds data products and publishes them to the rest of the organization. This is in contrast to the overworked Master Data Management team trying to model the whole business in a data warehouse based on their own with their generalist knolwedge of every domain. Worst of all, because all analytics go through them, they become a bottleneck and single point of failure for the organization. Data meshes distribute the work to those who know it best, and data products allow analysts and AI agents across the organization to query them through standardized interfaces. You can combine Data Mesh with [Medallion Architecture](https://www.databricks.com/blog/what-is-medallion-architecture-article) to take things to the next level. Medallion features three kinds of data: - Bronze for raw ingest, the source for any data analysis - Silver for clean, row-level refinement - Gold for curated aggregations reflecting common queries Each business domain in a mature data architecture offers Broze, Silver, and Gold data products, and most importantly, they all enjoy robust data governance, which provides semantic meaning across products. In other words, governance expresses metadata as glossaries, catalogs, and ontologies in natural language. This is critical context for Enterprise AI to add value, and it is often called the "Knowledge Layer." AI agents can query the Knowledge Layer for context about the underlying data sources and then reach into the data sources themselves for specific answers. This is progressive disclosure at enterprise scale, but you need the supporting architecture to make it work. There are also open source and commercial vendors out there offering ways to derive existing data into new projections to augment the Knowledge Layer, for example by creating graphs across different data sources. Whatever approach you take, just keep in mind that Knowlege Layers are critical for agents to drive value at work. I should also mention that making context discoverable and digestible by agents is starting to become so important that it is driving a whole new set of talking points around "Agent Experience," even going so far as to claim that Agent Experience (AX) is more important than user experience (UX). I would be cautious about this latest hype train. This is the most important reason: > Nothing will ever be more important than user experience becuase no one will ever be more important than users. The purpose of AX is to improve UX. AX also is merely the evolution of systems interoperabilty, something software engineers have thought about for decades. The vision for agent swarms acting autonomously is cool, but it doesn't mean discoverability, performance, resilency, and other key principles of systems interoperability are brand new ideas. To be fair, there is an art to AX that is new because now AI agents are making decisions rather than `if` statements in code. After all, the premise for this post is the importance of managing context to make your data available in the most usable way for agents, but I see AX as more buzzword than substance. Tech sure likes its buzzwords. Speaking of which... ## Context and Progressive Disclosure are the Antidote to Tokenmaxxing Like most Silicon Valley fads, tokenmaxxing, the act of maximizing AI token spend as a signal of productivity and innovation, was always completely unserious and frankly embarrasing. Our goal should be to build cool software that makes the lives of our users easier and better, not to indulge ourselves in AI for the sake of it and an unwillingness to seek therapy. [Uber](https://www.businessinsider.com/uber-coo-andrew-macdonald-ai-token-spending-harder-justify-2026-5) and [Amazon](https://www.404media.co/amazon-shuts-down-internal-ai-leaderboard-after-employees-cheated/), one of the worst offenders in Big Tech, finally figured it out. On the bright side, if you have had trouble making AI work for you, maybe it helps to know that industry leaders who should know better are so easily susceptible to hype and nonsense that even they [fail miserably](https://www.youtube.com/watch?v=F6uUY-BNLbY). If you want to use AI in a mature, serious way, you should consider tokens a precious resource like memory, processing power, and network bandwidth. This will save on costs to you and to the planet. Token efficiency starts with context. AI that veers off on its own without any guardrails wastes tokens. Context grounds agents and steers them in the right direction. However, too much context also wastes tokens. Combining context with progressive disclosure to control how agents work and guide them to the light at the end of the tunnel will maximize token efficiency. It's easy to be overwhelmed by all the new models, tools, techniques, and terms that emerge literally every day in the AI space. The anxiety is real. Rather than get caught up in hype, do yourself a favor. Focus on patterns around ways of working that transcend trends. Create a Culture of Context for yourself and your business to take the lead in AI. --- If you are still wondering, Thor's mother is [Queen Frigga of Asgard](https://marvelcinematicuniverse.fandom.com/wiki/Frigga). --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Infrastructure as Code Has a New Playbook: formae and Pkl" description: "formae brings agentic, stateless IaC with Pkl's type-safe config to AWS. No drift, no state files, and an MCP server AI agents can actually use." canonical: https://www.vidyasource.com/blog/infrastructure-as-code-formae-pkl-aws/ type: article published: 2026-03-14 author: "Neil Chaudhuri" tags: ["AI", "MCP", "Open Source", "DevSecOps", "Software Engineering"] image: https://www.vidyasource.com/img/blog/iac.webp --- # Infrastructure as Code Has a New Playbook: formae and Pkl > formae brings agentic, stateless IaC with Pkl's type-safe config to AWS. No drift, no state files, and an MCP server AI agents can actually use. - Canonical page: https://www.vidyasource.com/blog/infrastructure-as-code-formae-pkl-aws/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/infrastructure-as-code-formae-pkl-aws.json - Published: 2026-03-14 - Author: Neil Chaudhuri - Topics: AI, MCP, Open Source, DevSecOps, Software Engineering I have always been a big fan of "<Insert artifact here> as Code" implementations for several reasons. The artifacts are co-located with the code so you don't have to go hunting for them you need information. They also enjoy the same rigor like versioning and testing as production code. There are many great examples. Architecture as Code. Diagrams as Code. Documentation as Code. And of course the ubiquitous Infrastructure as Code (IaC). IaC is powerful because it allows you to automate the provisioning and management of your infrastructure through configuration files. Versioning across deploys and repeatability across environments are particularly useful, and the confidence to tear down and recreate entire environments on demand was a game change when IaC arrived on the scene. There are many popular IaC tools out there: - Terraform by HashiCorp, which uses a declarative language called HCL to manage multi-cloud resources - OpenTofu, the open source fork of Terraform - AWS CloudFormation, Amazon's native IaC service for provisioning AWS resources via JSON or YAML templates - Ansible by Red Hat, an agentless automation tool for configuration management and orchestration - Pulumi, which lets you define infrastructure using general-purpose programming languages like TypeScript, Python, and Go While all of these represent significant advancements over the status quo before IaC, there were several significant problems. Configuration drift is the slow, silent disaster of infrastructure management. Someone updates an S3 bucket policy in the console. A teammate tweaks an ECS task definition directly. Nobody writes it down. Six months later your code says one thing and production says something completely different, and you spend two days figuring out why your deployment broke. The root cause is almost always the same: configuration files have no types. YAML, JSON, and TOML are all happy to accept `maxConnections: "fifty"` right alongside `maxConnections: 50`. There are just magic strings. As someone obsessed with type safety, I find this especially problematic. While Pulumi solves this particular problem by replacing endless walls of configuration text with Turing complete, probably typed programming languages, this makes configuration more complex, less deterministic, and more error-prone because you have introduced the possibility of bugs and side effects just like your production code. In other words, this approach overcompensates too far in the opposite direction. All of this means there is room a for a happy medium that can offer the best of both worlds. Two tools have combined to reshape what IaC can actually look like. One is [Pkl](https://pkl-lang.org/), Apple's open source, type safe configuration (but not programming) language. You can use it anywhere you need configuration, from Spring Boot to Kubernetes. The other is [formae](https://github.com/platform-engineering-labs/formae) (yes, spelled in lowercase like poet ee cummings or musician k.d. lang), a new IaC tool that uses Pkl for configuration files that represent the source of truth for your infrastructure. Together they plug the leaks that have made traditional IaC so frustrating. formae also goes one step further with an MCP server that lets AI agents reason about and act on your infrastructure directly. Let's dig in. ## Pkl: Configuration Errors Are Now Compile-Time Errors For such a simple language, Pkl is really quite powerful. It offers typed properties, constraints, union types, default values, inheritance through `amends`, and computed properties. When your config violates a constraint, you get an error *at evaluation time*, before anything gets deployed. It's the "make impossible states impossible" principle I love so much applied to infrastructure. Here's what that looks like in practice. Imagine you're defining the configuration for your ECS task definition shared across environments: ```pkl // infra/base/EcsTaskConfig.pkl module EcsTaskConfig typealias CpuUnits = Int(this == 256 || this == 512 || this == 1024 || this == 2048 || this == 4096) typealias MemoryMiB = Int(this >= 512 && this <= 30720) // Only allow known, safe log levels — no "YOLO" in production typealias LogLevel = "DEBUG" | "INFO" | "WARN" | "ERROR" class ContainerConfig { image: String(startsWith("123456789012.dkr.ecr.us-east-1.amazonaws.com/")) cpu: CpuUnits = 512 memory: MemoryMiB = 1024 logLevel: LogLevel = "INFO" portMappings: Listing } class EcsTaskConfig { family: String containers: Listing(length >= 1) executionRoleArn: String(startsWith("arn:aws:iam::")) taskRoleArn: String(startsWith("arn:aws:iam::")) } ``` This is like defining classes in Object-Oriented Programming. Then your production config `amends` this base and fills in the specifics. Think of it like instantiating the classes you defined. ```pkl // infra/production/ecs-webapp.pkl amends "package://pkg.pkl-lang.org/github.com/platform-engineering-labs/formae/formae@0.15.0#/Resource.pkl" import "../base/EcsTaskConfig.pkl" family = "webapp-production" executionRoleArn = "arn:aws:iam::123456789012:role/ecsTaskExecutionRole" taskRoleArn = "arn:aws:iam::123456789012:role/webappTaskRole" containers { new EcsTaskConfig.ContainerConfig { image = "123456789012.dkr.ecr.us-east-1.amazonaws.com/webapp:latest" cpu = 1024 memory = 2048 logLevel = "INFO" portMappings { 8080 } } } ``` Try setting `cpu = 999` or `logLevel = "TRACE"` or an image URI that doesn't match your ECR registry. Pkl tells you immediately. You've moved configuration errors from "3 AM production incident" to "the moment you save the file." That's the kind of feedback loop shift that DORA research has confirmed makes teams more productive. Take a look at the `typealias` called `CpuUnits`. AWS ECS only accepts specific CPU values. The Pkl defintinion above encodes that business rule directly in the type as a union of type of a finite set of integer values. You can't accidentally set `cpu = 300`. The type system prevents it. This is what statically typed configuration means in practice, and it's why I'm a big fan of Pkl. ## formae: IaC That Treats Code as Truth Terraform and OpenTofu are the most popular IaC tools. They manage *state files* that track what's deployed but they can become a liability at scale. They need to be stored somewhere, typically somewhere remote, with their own locking mechanism. The big problem is that it is common for someone to alter the infrastructure manually in the console because they do not recognize state files as the source of truth. With the state files now misaligned with the real infrastructure, this is textbook configuration drift. Terraform won't know it until you make the effort to run `terraform plan` periodically as a chrom job or in CI/CD on a schedule or on an event basis (like a pull request). Running `terraform apply` to make changes will also detect drift. Still, in both cases, it's on you to figure it out. formae has no state files. Your Pkl configuration *is* the source of truth. formae's agent (not an AI agent but rather a long-running daemon with its own HTTP server, datastore connection, and retry logic) continuously reconciles reality against your code. If something drifts outside your IaC definitions, like a manual change in the AWS console or an automated key rotation, formae detects it and can merge or reject the change depending on your configuration. This is pretty cool. Consider a simple web application deployed to AWS. Let's see what it looks like to manage an S3 bucket for static assets alongside an ECS-hosted web application behind CloudFront. First, configure your AWS target using formae's Pkl types: ```pkl // infra/formae-config.pkl amends "package://pkg.pkl-lang.org/github.com/platform-engineering-labs/formae/formae@0.15.0#/Config.pkl" local awsTarget = new Target { label = "aws-production-us-east-1" namespace = "AWS" discoverable = true config { type = "aws" region = "us-east-1" profile = "production" } } targets { awsTarget } ``` Then define your stack on AWS: - S3 bucket for assets - ECS for the app - CloudFront as the CDN for both ```pkl // infra/production/webapp-stack.pkl amends "package://pkg.pkl-lang.org/github.com/platform-engineering-labs/formae/formae@0.15.0#/Stack.pkl" // --- S3: Static Asset Bucket --- local assetBucket = new Resource { label = "webapp-assets" type = "AWS::S3::Bucket" target = "aws-production-us-east-1" stack = "webapp-production" properties { BucketName = "my-webapp-assets-prod" VersioningConfiguration { Status = "Enabled" } PublicAccessBlockConfiguration { BlockPublicAcls = true BlockPublicPolicy = true IgnorePublicAcls = true RestrictPublicBuckets = true } Tags { ["Environment"] = "production"; ["ManagedBy"] = "formae" } } } // --- ECS Cluster --- local ecsCluster = new Resource { label = "webapp-cluster" type = "AWS::ECS::Cluster" target = "aws-production-us-east-1" stack = "webapp-production" properties { ClusterName = "webapp-production" Tags { ["Environment"] = "production" } } } // --- ECS Service --- local ecsService = new Resource { label = "webapp-service" type = "AWS::ECS::Service" target = "aws-production-us-east-1" stack = "webapp-production" properties { ServiceName = "webapp-production-service" Cluster = ecsCluster.properties["ClusterName"] TaskDefinition = "webapp-production:latest" DesiredCount = 2 LaunchType = "FARGATE" NetworkConfiguration { AwsvpcConfiguration { AssignPublicIp = "DISABLED" Subnets { "subnet-abc123"; "subnet-def456" } SecurityGroups { "sg-webapp-prod" } } } Tags { ["Environment"] = "production"; ["ManagedBy"] = "formae" } } } // --- CloudFront Distribution --- local cdn = new Resource { label = "webapp-cdn" type = "AWS::CloudFront::Distribution" target = "aws-production-us-east-1" stack = "webapp-production" properties { DistributionConfig { Enabled = true PriceClass = "PriceClass_100" HttpVersion = "http2and3" Origins { // S3 origin for static assets new { Id = "S3-webapp-assets" DomainName = "\(assetBucket.properties["BucketName"]).s3.us-east-1.amazonaws.com" S3OriginConfig { OriginAccessIdentity = "origin-access-identity/cloudfront/ABCDEFG" } } // ECS / ALB origin for the app new { Id = "ALB-webapp" DomainName = "webapp-prod-alb-1234567890.us-east-1.elb.amazonaws.com" CustomOriginConfig { HTTPSPort = 443 OriginProtocolPolicy = "https-only" } } } DefaultCacheBehavior { TargetOriginId = "ALB-webapp" ViewerProtocolPolicy = "redirect-to-https" CachePolicyId = "658327ea-f89d-4fab-a63d-7e88639e58f6" // CachingOptimized } CacheBehaviors { new { PathPattern = "/assets/*" TargetOriginId = "S3-webapp-assets" ViewerProtocolPolicy = "redirect-to-https" CachePolicyId = "658327ea-f89d-4fab-a63d-7e88639e58f6" } } } } } resources { assetBucket; ecsCluster; ecsService; cdn } ``` Now apply your configuration in the command line (probably in your CI/CD pipeline). ```bash formae apply infra/production/webapp-stack.pkl --target aws-production-us-east-1 ``` formae's agent picks up your configuration, resolves the dependencies in the right order (S3 before CloudFront, cluster before service), and drives the changes asynchronously. If you like, you can even simulate your changes first with `--simulate` before you make things official. Most importantly, the formae agent checks periodically for drift. If someone manually changed your S3 bucket policy or scaled the ECS service through the console, formae validates that the change conforms to the rules you defined through your type safe Pkl configuration. If it does, it incorporates the changes. If it doesn't, formae tells you. If formae existed a few years ago, I would have thought its automated drift detection and validation was its coolest feature. But in the age of AI, there is something else formae can do that is probably cooler. ## The formae MCP Server: Your AI Agent Can Now Manage Infrastructure formae ships with an [MCP server](https://modelcontextprotocol.io/), so you can use any MCP client like Goose, Claude Desktop, or any modern IDE to query and, if your organizational policy allows, update your infrastructure directly using natural language. That is pretty cool. When you run the formae agent and expose its MCP server, you can configure your MCP client to do things like this: - **Query your inventory** — Ask "What S3 buckets are in the production stack?" and get back the right answer in real time. - **Check drift** — Ask "Has anything changed in the webapp-production stack since the last check?" if you want to check for configurtion drift manually. - **Simulate changes** — Ask "What would change if I applied this config?" before any real action is taken. - **Apply infrastructure** — Let an agent propose and apply a formae config with human-in-the-loop approval. This is already awesome, but imagine the possibilities. For example, AI agents could connect to formae MCP as part of a swarm that is also connected GitHub, Jira, and your entire engineering infrastructure for autonomous action. Of course you need a mature AI governance and guardrail strategy to make that work, but that is a story for another day. The main point here is formae MCP opens up a lot of options. This is the right model for AI + infrastructure. A live, type-validated, code-first source of truth with an MCP interface that AI agents can actually trust and act on. ## Why The Pkl-formae Combination Works Pkl and formae reinforce each other in a way that's more than the sum of their parts. Pkl gives you the type system. Types enforce your infrastructure. Constraints enforce your business rules. The `amends` feature enable reuse so staging and production share a common base without copy-paste drift. formae gives you the runtime guarantee. It uses Pkl to validate your configuration. It validates and detects drift for you. It manages dependencies among resources. Its MCP server enables AI to understand your infrastructure—not as a novelty but as a genuine operational, natural language interface. Your AI assistant knowing in real time how many ECS services are running, whether any resources have drifted, and what a proposed change would actually do to your infrastructure is powerful. That's AI at its best in an organization. The old playbook of raw YAML, state files in S3 with DynamoDB locking, CI/CD running `terraform plan` on a schedule, and crossing your fingers got us pretty far, but configuration drift, fragile state, and zero feedback at definition time have always been cracks in the foundation. Pkl seals them at the config layer. formae seals them at the infrastructure layer. Together, they represent Infrastructure as Code at its modern best. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Python Is 31x Slower: You Can Do Better for Enterprise AI" description: "A new MCP server benchmark shows Python is 31x slower than Go and Java—reinforcing why enterprise AI demands languages built for integration and performance, not just experimentation." canonical: https://www.vidyasource.com/blog/python-is-31x-slower-new-benchmark-confirms-not-language-for-enterprise-ai/ type: article published: 2026-02-16 author: "Neil Chaudhuri" tags: ["AI", "MCP", "Python", "Java", "Machine Learning", "Programming", "Architecture"] image: https://www.vidyasource.com/img/blog/ai-mcp-benchmark.webp --- # Python Is 31x Slower: You Can Do Better for Enterprise AI > A new MCP server benchmark shows Python is 31x slower than Go and Java—reinforcing why enterprise AI demands languages built for integration and performance, not just experimentation. - Canonical page: https://www.vidyasource.com/blog/python-is-31x-slower-new-benchmark-confirms-not-language-for-enterprise-ai/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/python-is-31x-slower-new-benchmark-confirms-not-language-for-enterprise-ai.json - Published: 2026-02-16 - Author: Neil Chaudhuri - Topics: AI, MCP, Python, Java, Machine Learning, Programming, Architecture Last year I wrote about why [Python is not the language of AI](https://www.vidyasource.com/blog/python-is-not-language-of-ai/): While Python dominates machine learning, its strengths for fast, lightweight experiments become liabilities when you transition to the context integration and performance challenges that define enterprise AI. The response was enormous and the debate was spirited. Now we have hard numbers to back it up. ## The Benchmark TM Dev Lab has published an [MCP Server Performance Benchmark](https://www.tmdevlab.com/mcp-server-performance-benchmark.html) comparing Python, Go, Java, and TypeScript across key performance dimensions. [Model Context Protocol](https://modelcontextprotocol.io/) is the [AAIF standard](blog/biggest-news-in-ai-since-anthropic-mcp) for connecting AI systems to enterprise data and tools. As I noted in my original post, AI is fundamentally a *context integration challenge*, and MCP servers are the infrastructure that makes enterprise AI agents possible at scale. Their performance is critical given how often agents will use them. The benchmark's conclusion about Python is blunt: > Not Recommended For: Any production high-load scenario (31x slower than Go/Java) Thirty-one times slower. Not 31% slower. *Thirty-one times.* ## The Results The two languages in the benchark built for enterprise rigor, Java and Go, delivered exactly as you'd expect: ![Bar chart comparing average latency in milliseconds across four MCP server implementations over three rounds. Java and Go are nearly identical at 0.84ms and 0.86ms respectively. Node.js averages 10.66ms, and Python is slowest at 26.45ms — roughly 31 times slower than Java and Go](https://www.vidyasource.com/img/blog/ai-mcp-benchmark.webp) Go edged out Java overall, but organzations don't operate in a vacuum. Most medium-to-large enterprises have significantly more Java infrastructure and institutional knowledge than Go. Given that reality, Java represents the smarter tradeoff for most organizations. But if you have a strong Go team? Even better. ## Why This Shouldn't Surprise Anyone As I wrote before, Python has qualities that make it extraordinary for data science and machine learning: dynamic typing that lets data scientists experiment quickly, an intuitive syntax that doesn't demand software engineering ceremony, and a prolific ecosystem of libraries from Pandas and NumPy to PyTorch and TensorFlow. But those same qualities work against you in production: - **Lack of type safety** — This is critical for grounding AI systems in structure. Even the Python AI community recognizes this, which is why libraries like Pydantic AI and BAML exist to retrofit type safety onto a language that doesn't natively provide it. - **Poor performance at scale** — FastAPI and other modern libraries address this in limited scope, but as this benchmark demonstrates, the fundamental performance gap remains enormous. - **Lack of mature middleware** — Enterprise AI demands robust Kafka integration, message queues, and event-driven architectures that the Python ecosystem doesn't serve well. - **Limited tooling for security, observability, and resiliency** — These are non-negotiable concerns of production software. The MCP benchmark quantifies what experienced enterprise architects already know intuitively. When you're building the connective tissue between LLMs and enterprise data—databases, APIs, event-driven pipelines, and SaaS platforms like Salesforce and Confluence—Python simply isn't built for the job. ## AI Is a Context Integration Challenge This bears repeating because it's the fundamental insight that drives everything else: > AI success is a function of context not computation. Python makes computations like clustering and loss functions easy, but building with AI means building agents that can harvest your institutional memory across the enterprise as context for LLMs. It's an integration challenge. It's API calls, not math, operating under the constraints of enterprise security, observability, and performance requirements. Software engineers have been solving these problems for decades with languages and frameworks purpose-built for the task. MCP servers are a concrete manifestation of this. They're the infrastructure that connects AI to your enterprise context. When that infrastructure is 31x slower than it needs to be, you're not just wasting compute. You're throttling the quality and responsiveness of every AI capability that depends on it. ## The Right Tools Exist As I outlined in my original post, there are languages whose features, runtimes, and ecosystems make them far better options for enterprise AI: - **Java and Kotlin** on the JVM, the CPU leader according to this benchmark featuring frameworks like [Koog](https://github.com/JetBrains/koog) and [Embabel](https://github.com/embabel/embabel-agent) that embrace type safety, testing, Domain-Driven Design, and observability - **Go**, built for speed and concurrency, now validated by this benchmark as the memory leader and overall performance leader - **TypeScript** with frameworks like [Mastra](https://mastra.ai/) and mature integration libraries like Prisma and tRPC - **C# and F#** on .NET with [Semantic Kernel](https://github.com/microsoft/semantic-kernel) These frameworks affirm the lessons we've learned about mature software engineering at scale. Type safety, testing, security, observability, and disaster recovery aren't optional. They're table stakes for serious software deployed at scale, AI or otherwise. ## Some Caveats The usual caveats apply. Benchmarks are never the full story. MCP servers are only one segment of AI deployments. Implementation details, team expertise, and organizational context all factor into technology decisions. But when the performance gap is measured in *orders of magnitude*, it should give you pause. If you're deploying Python as the backbone of your enterprise AI infrastructure, this benchmark is one more data point suggesting you can do better. Succeeding with AI begins with sound business strategy and data strategy. After that, your technical implementation is a matter of integrating context, not performing computations. Solve the right problem with the right solution. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "The Biggest News in AI Since MCP" description: "The Linux Foundation launches the Agentic AI Foundation, which formalizes MCP, Goose, and AGENTS.md into global open standards." canonical: https://www.vidyasource.com/blog/biggest-news-in-ai-since-anthropic-mcp/ type: article published: 2025-12-12 author: "Neil Chaudhuri" tags: ["AI", "Software Engineering", "Programming"] image: https://www.vidyasource.com/img/blog/aaif.webp --- # The Biggest News in AI Since MCP > The Linux Foundation launches the Agentic AI Foundation, which formalizes MCP, Goose, and AGENTS.md into global open standards. - Canonical page: https://www.vidyasource.com/blog/biggest-news-in-ai-since-anthropic-mcp/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/biggest-news-in-ai-since-anthropic-mcp.json - Published: 2025-12-12 - Author: Neil Chaudhuri - Topics: AI, Software Engineering, Programming When I spoke at the [MCP Roundtable](https://www.vidyasource.com/blog/vidya-talks-ai-and-mcp/), one of the points I made is that MCP wouldn't really hit until it "grew up" by becoming a real standard. HTTP, HTML, JSON, and other standards changed the world only because of the W3C, a global consortium transcending vendors and corporate strategies with a grand vision for the web. That, and more, has now happened. ## The News The Linux Foundation has [announced](https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation) formation of the Agentic AI Foundation (AAIF), a neutral, open-governance initiative designed to accelerate the evolution of Agentic AI. AAIF unites some of the leading industry players with the leading open technologies—MCP from Anthropic, Goose (which I also discussed at the MCP Roundtable and use regularly) from Block, and AGENTS.md from OpenAI—into a unifying vision for agents. Why does this matter? - AAIF establishes shared, vendor-neutral standards for building and coordinating AI agents. - AAIF breaks down proprietary silos to enable interoperability across tools, platforms, and organizations. - AAIF shifts Agentic AI from fragmented experiments to a more unified, production-ready ecosystem. This is exciting. It's wonderful finally to see global, open standards for building AI agents. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Vidya Talks AI and MCP at NOVA SART" description: "Key insights from our appearance on the MCP Roundtable at NOVA SART: MCP's promise and peril, AI architecture, security, real-world use cases, and why AI will increase demand for software expertise." canonical: https://www.vidyasource.com/blog/vidya-talks-ai-and-mcp/ type: article published: 2025-09-08 author: "Neil Chaudhuri" tags: ["AI", "MCP", "Programming", "Testing", "Architecture", "Observability", "Open Source", "Security"] image: https://www.vidyasource.com/img/blog/mcp-panel.webp --- # Vidya Talks AI and MCP at NOVA SART > Key insights from our appearance on the MCP Roundtable at NOVA SART: MCP's promise and peril, AI architecture, security, real-world use cases, and why AI will increase demand for software expertise. - Canonical page: https://www.vidyasource.com/blog/vidya-talks-ai-and-mcp/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/vidya-talks-ai-and-mcp.json - Published: 2025-09-08 - Author: Neil Chaudhuri - Topics: AI, MCP, Programming, Testing, Architecture, Observability, Open Source, Security - Video: https://www.youtube.com/watch?v=VXoeIWZoxBU It was an honor for me to participate in the Model Context Protocol Roundtable at the [NOVA SART Meetup](https://www.meetup.com/nova-architecture-round-table-meetup/) hosted by Solution Street. I had so much fun sharing and gaining insights and meeting wonderful people as we all try to make sense of how AI and MCP are shaping our own personal experiences as well as application architecture in the enterprise. Thanks to Solution Street's Katie Schuman for organizing everything and to Gregory Hodum for leading the discussion. Thanks also to Soujanya Vullam from Yahoo and Pervez Choudhry from Gentoro for being great fellow panelists and for teaching me so much. And of course thanks to everyone who attended for their attention and stimulating questions. We covered a lot of ground including the following: - MCP primitives like clients, servers, prompts, and resources - Security, observability, type safety, Dome Driven Design, and other software engineering disciplines that are critical for AI - Why AI will not take your job and in fact will demand even more software expertise - Real-world use cases for MCP - MCP tools you can use today for free to improve life at work and home It was a really fun discussion. Check it out below and [get in touch](https://www.vidyasource.com/contact/) if you want to talk more! --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Python Is Not the Language of AI" description: "Python is the language of machine learning and training AI models, but there are several reasons why Python is not the language of AI." canonical: https://www.vidyasource.com/blog/python-is-not-language-of-ai/ type: article published: 2025-09-07 author: "Neil Chaudhuri" tags: ["AI", "MCP", "Python", "Java", "Machine Learning", "Kotlin", "Programming", "Testing", "Architecture", "Open Source", "Observability", "Salesforce", "Hugging Face"] image: https://www.vidyasource.com/img/blog/python.webp --- # Python Is Not the Language of AI > Python is the language of machine learning and training AI models, but there are several reasons why Python is not the language of AI. - Canonical page: https://www.vidyasource.com/blog/python-is-not-language-of-ai/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/python-is-not-language-of-ai.json - Published: 2025-09-07 - Author: Neil Chaudhuri - Topics: AI, MCP, Python, Java, Machine Learning, Kotlin, Programming, Testing, Architecture, Open Source, Observability, Salesforce, Hugging Face MIT recently reported that a staggering [95% of enterprise AI deployments fail](https://www.forbes.com/sites/jaimecatmull/2025/08/22/mit-says-95-of-enterprise-ai-failsheres-what-the-5-are-doing-right/). That is *really bad!* The primary reasons are strategic rather than technical like the absence of a data strategy and no clear business objectives beyond "We need to use more AI or we will fall behind!" In all things, AI or otherwise, whatever you do downstream from terrible leadership is pretty much doomed to fail. However, even the most thoughtful strategy will fail without a very particular technical understanding: *Python has always been and will likely always be the language of machine learning (ML), but it is not the language of AI. The failure to understand the difference is hurting AI adoption. Succeeding with AI is a function of context not computation.* ## A little history Imagine you work at Netflix in 2010. You have a hypothesis: People are more likely to watch content, and therefore ads that will make your manager happy, that takes place in the city where they grew up. For example, someone who grew up in Scranton, Pennsylvania, is more likely to watch [The Office](https://www.imdb.com/title/tt0386676/) than they would otherwise. At that time and still to this day, Python was the best choice to test your hypothesis. With its dynamic typing and intuitive syntax, the Python language makes it very easy for data scientists, who are not programmers and understandably have no interest in the ceremony of traditional software engineering, to experiment quickly. With that growing popularity came the blossoming of a prolific ecosystem. Libraries we take for granted today like Pandas, NumPy, Scikit-learn, spaCy, GLiNER, and countless others emerged with Deep Learning staples like PyTorch and TensorFlow arriving later. Machine learning pipelines using tools like dbt, Dagster, Airflow, and Great Expectations became ubiquitous. Bombshell tooling like Jupyter Notebooks and powerful IDEs [entered the villa](https://www.youtube.com/watch?v=Ihum2Yk3Zyk). It is easy to see why the Python ecosystem has been so dominant for so long for running experiments and solving many popular use cases like spam detection, sentiment analysis, fraud detection, loan default prediction, and image classification. There are powerful data science alternatives like R and Julia, but they just don't have the juice. When ChatGPT came along in 2022, Python appeared poised to [become more powerful than you can possibly imagine](https://www.youtube.com/watch?v=iVBX7l2zgRw). With Python the clear choice for NLP and Deep Learning, AI companies use Python to train Large Language Models (LLMs). If you want to fine tune an existing model to train it for your use case, you use Python tools like [Unsloth](https://unsloth.ai/). Naturally, Python also led the way in AI orchestration. As Retrieval Augmented Generation (RAG) became all the rage, [LangChain](https://www.langchain.com/) and [LlamaIndex](https://www.llamaindex.ai/) inherited the Python pipeline tradition and made it easier to ground LLMs on your own structured and unstructured data. To take things even further, because Python has dominated ML for so long and there have always been so many outstanding ML models, and now AI models, on Hugging Face, core Hugging Face libraries like Transformers are Python libraries. Other approaches like [Deep Java Library](https://docs.djl.ai/master/docs/demos/huggingface/hybrid/index.html) and [Spring AI](https://docs.spring.io/spring-ai/reference/api/chat/huggingface.html) are technically possible to work with Hugging Face, but the process is never very straightforward. The result was predictable. You can see the surge in Python popularity in various Python language rankings like the August 2025 TIOBE Index, which not only puts Python at #1 but also jumping 8.1% after the last ranking! ![August 2025 TIOBE Index showing Python #1 and with a huge 8.1% jump from the previous ranking](https://www.vidyasource.com/img/blog/tiobe-python.webp) However, as AI adoption explodes, organizations are butting against the limits of Python for building mission-critical, production software for real users. Python doesn't readily deliver what enterprise AI needs to thrive. ## In the Real World, ML ≠ AI It's a vast oversimplification, but machine learning is computation. And to be fair, it's true you need very advanced computations to build LLMs in the first place. But once you have LLMs in place doing real work, generating value from them for your business is not about computation at all. It's about *context*. The key to success with AI is to provide the best context to the best suited LLM(s) so they can pattern match their way to the best result. For you individually, if you are using an LLM to find a job, context means using [Model Context Protocol (MCP)](https://modelcontextprotocol.io/docs/getting-started/intro) clients like [Goose](https://block.github.io/goose/) to integrate the job description on a company website and other relevant pages about company strategy and values with the PDF resume on your laptop. That context helps LLMs offer suggestions on what to emphasize in your work history to maximize your chances for an interview. That doesn't require any code from you, Python or otherwise. It's simply knowing which context matters and making it available to AI. Similarly for a business, context means gleaning the most relevant information across the enterprise—in databases, Notion, Jira, Slack, GitHub, Salesforce, Dropbox, Confluence, SharePoint, Google Workspace, Office 365, and all the other typical black holes of organizational memory in big companies—to your problem and providing *that* to an LLM. This likely requires some code, but it's just API calls to the data and the LLM(s), which software engineers have been doing forever, rather than calculating [root mean squared error](https://statisticsbyjim.com/regression/root-mean-square-error-rmse/). Put simply, AI is a context integration challenge not a number crunching challenge. ## Python Limitations I have a deep appreciation for Python. In fact, it's the language I recommend to people who want to get into programming for the first time for all the same reasons it's so popular with data scientists. But there are several reasons why Python never really caught on to solve integration challenges at scale in enterprises: - Lack of type safety, which [Pydantic AI](https://ai.pydantic.dev/), [BAML](https://boundaryml.com/), and other modern AI libraries try to address because it's so important to ground LLMs in structure - Poor performance of Python at scale, which [FastAPI](https://fastapi.tiangolo.com/) and other modern libraries address in limited scope - Lack of mature middleware options like Kafka integration - Lack of mature tooling around security, observability, and resiliency There is an old cliché in software engineering: Use the right tool for the job. The qualities that make Python dominant for ML are irrelevant, even counterproductive, to what you need for AI. Python is fine, but it's not the right tool for the job. ## Better Choices for AI If AI is an integration challenge rather than an ML challenge, AI success demands technologies built to solve integration challenges. There are lots of languages whose features, runtimes, and ecosystems make them far better options than Python: - JVM languages like Java, which has dominated the enterprise for decades, and Kotlin, which is a modern alternative interoperable with Java and currently my favorite language - .NET languages like C#, which also dominates the enterprise, and F# - TypeScript, which has emerged very quickly as a popular integration option with libraries like Prisma and tRPC - Go, built for speed and integrates well with middleware - Rust, *really* built for speed with a devoted, growing community willing to tackle the tough learning curve - Elixir, niche language but purpose built for scalable, fault tolerant distributed systems Your choice depends on all the usual considerations like available talent and skills, deployment constraints, available SDKs, and so on. The key point is you are not limited to Python for AI because of its legacy in ML. You can even do better. You will find higher-level agent frameworks that rival and arguably beat LangChain and LlamaIndex like [Koog](https://docs.koog.ai/) (Kotlin) and [Embabel](https://docs.embabel.com/embabel-agent/guide/0.1.2-SNAPSHOT/) (Kotlin and Java) on the JVM, [Mastra](https://mastra.ai/) in TypeScript, and [Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/overview/) in C#, Java, and yes, Python. The great thing about these frameworks is they affirm the lessons we have learned about mature software engineering at scale over the last few decades by espousing concepts like type safety, testing, security, Domain-Driven Design, observability, and disaster recovery. While completely unnecessary for data science experiments, these concepts are all crucial for serious software deployed at scale, AI or otherwise. Succeeding with AI begins with sound business strategy and data strategy. After that, your technical implementation is a matter of integrating context not performing computations. Solve the right problem with the right solution to maximize your chances for success with AI. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Beyond GitHub Culture: Why Your Team Should Reconsider Pull Requests" description: "Pull requests make sense in open-source but might be slowing your internal team down. Explore some more efficient alternatives." canonical: https://www.vidyasource.com/blog/beyond-github-culture-reconsider-pull-requests/ type: article published: 2025-06-09 author: "Neil Chaudhuri" tags: ["Agile", "AI", "Open Source", "Software Engineering", "Programming", "Testing", "Continuous Integration", "DevSecOps"] image: https://www.vidyasource.com/img/blog/pair-programming.webp --- # Beyond GitHub Culture: Why Your Team Should Reconsider Pull Requests > Pull requests make sense in open-source but might be slowing your internal team down. Explore some more efficient alternatives. - Canonical page: https://www.vidyasource.com/blog/beyond-github-culture-reconsider-pull-requests/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/beyond-github-culture-reconsider-pull-requests.json - Published: 2025-06-09 - Author: Neil Chaudhuri - Topics: Agile, AI, Open Source, Software Engineering, Programming, Testing, Continuous Integration, DevSecOps Someone on LinkedIn was lamenting how the pull request (PR) and code review policies of product teams often slow them down. This reminded me of the old Henny Youngman joke: "The patient says, 'Doctor, it hurts when I do that.' The doctor says, 'Then don't do that!'" Maybe if PRs hurt, don't do them? To understand the problem with PRs, we need to talk about gates and waterfall software development. ## A short history lesson In the old days before TikTok and wifi, teams used to develop software using a method called "waterfall," which was based on construction—the way you erect buildings. When applied to software, waterfall began with lengthy, meticulous planning by systems engineers in an attempt to gather every possible requirement and anticipate every possible risk. After months or even years, they'd finish and reach the first gate. They would toss the deliverable, a requirements book the size of a Tolstoy novel, over to the development team. They'd write a bunch of code, maybe test it on their machines, and hit the next gate. They would toss their code to the Quality Assurance (QA) team to test. Because project managers wouldn't stand for idling, the developers would of course keep working, layering new code on the previous, likely buggy code while the testers tested the batch they just got. The testers would finish, find a bunch of bugs, hit the next gate, and toss their findings back to the developers, who would fix them. This took longer than it had to because all the new code added since made it harder to debug the problems with the original code. Anyway, this would go back and forth until eventually the code was good enough to real the final gate. The operations team would take the code and deploy it, probably several years after the initial requirements gathering. At this point, I have no doubt you are curled up in a fetal position under a weighted blanket listening to ASMR. Waterfall was a terrible way to build software for a million reasons. The biggest one might be all those gates and handoffs from one team to the next. They make batches bigger and feedback slower, which is the worst thing you can do in a creative field demanding a lot of experimentation and fast feedback like software engineering. Agile and DevSecOps recognized these problems and saved us. But not completely. ## The problem with PRs Fast forward to today. Many of us who write code for a living were weaned on GitHub and open-source where PRs and code reviews are second nature. But think about what PRs really are. PRs are modern gates, and gates, past and present, slow you down. You need to wait for a reviewer to find the time amidst their other duties to review your code. There will be a back and forth over the comments. Meanwhile, others on your team are submitting PRs from their own branches, which could very easily conflict with your pending PR. Things can get really messy with merge conflicts, especially since most teams lack the discipline to keep PRs small enough to complete in a day or less. On the other hand, while software engineering is a creative endeavor where we should avoid gates, it is also about tradeoffs. Rather than wallow in dogma, we must always be willing to bend our rules when it's to our advantage. PRs are gates that slow you down, but they're necessary in open-source because open-source is a low-trust environment. You can't let just anyone submit the untested code they wrote while watching [Rick and Morty](https://www.adultswim.com/videos/rick-and-morty) to the code base and disrupt everyone else. The problem is most organizations have not thought about whether we have to pay the same price in a *high*-trust environment like our own product teams inside the business. The answer is probably not. In fact, [DORA has found that the most productive teams use a Trunk-Based Development approach](https://dora.dev/capabilities/trunk-based-development/), where developers make changes to the code base (often labeled "main" or "trunk") directly by leveraging [branch by abstraction, feature flags, and dark launches](https://www.youtube.com/watch?v=pXovk-5J0Lg) that Dave Farley and Martin Fowler have promoted for years. Meanwhile, reviews are done as you build through test and static analysis automation and maybe pair programming. ## Pair programming Pair programming may not come naturally (It doesn't for me!), and there are always factors to consider like how often to pair, how long to pair in one sitting, and most importantly, how to accommodate teammates who are self-conscious, neurodivergent, or however uncomfortable. Still, if you are willing to take the time to ask Stack Overflow or Reddit or AI for help, maybe consider asking your teammate next door to sit with you for a few minutes instead. There are [considerable benefits](https://martinfowler.com/articles/exploring-gen-ai/05-not-your-pair-programmer.html). You both can also add AI to help with your review. Whichever approach you take to pair programming, if you don't love it, then you don't love it. In the end, please at least consider the possibility that what works best in the Wild West of open-source may not be what works best in your organization. At a time when tech obsesses over productivity and AI, it's long past time to consider new ways of working that make us better. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "It's Not About Unit Or Integration Tests. It's About Confidence" description: "Forget traditional testing categories and focus on what builds confidence. Use AI to generate test data efficiently." canonical: https://www.vidyasource.com/blog/not-about-unit-integration-tests-about-confidence/ type: article published: 2025-05-30 author: "Neil Chaudhuri" tags: ["Agile", "Java", "Spring Boot", "Software Engineering", "Programming", "Testing", "Continuous Integration", "JUnit"] image: https://www.vidyasource.com/img/blog/testing-microscope.jpg --- # It's Not About Unit Or Integration Tests. It's About Confidence > Forget traditional testing categories and focus on what builds confidence. Use AI to generate test data efficiently. - Canonical page: https://www.vidyasource.com/blog/not-about-unit-integration-tests-about-confidence/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/not-about-unit-integration-tests-about-confidence.json - Published: 2025-05-30 - Author: Neil Chaudhuri - Topics: Agile, Java, Spring Boot, Software Engineering, Programming, Testing, Continuous Integration, JUnit If you write Java code, you probably know Dan Vega. His talks and YouTube channel have helped countless Java engineers get better. Dan recently offered some pointers for dealing with the challenges of integrating with a new API in your Spring Boot application. I found one of them particularly important. > Always test your JSON deserialization separately before diving into HTTP calls I couldn't agree more, but I think this is really one example of a broader point about testing your code efficiently. I have never cared for the "unit test" vs "integration test" distinction. For one thing, it's been like 20 years and we still can't agree on how to define them. But more importantly, I don't love defining a test by its architecture. What matters is its purpose. In other words, we should always be asking two important questions: * What is the thing I am worried about that this test helps me figure out? * What is the simplest and fastest way I can figure it out? In Dan's example, there is an issue with how the Spring Boot code processes the response from that new API call. There are multiple possible reasons that can happen. The first one is this: Does the shape of my data model (probably a JavaBean or Record) match the shape of the JSON response? Or more simply, am I actually getting back what I expect to be getting back? It's a deserialization question. There are a lot of ways to test this. Many devs would test against the real API. That incorporates the network when you don't need it and makes the test complicated to set up, slow to run, and unpredictable because you have introduced network side effects. You may also have to contend with rate limits on the API. Other devs might be a little more clever and use [Testcontainers MockServer](https://testcontainers.com/modules/mockserver/) to mock the API, but even that's overkill. After all, you don't need infrastructure of any kind. All you need is JSON. Use AI to model as many stub API responses as you want. The best way would be model them on an OpenAPI spec if the API makes one available. If not, prompt your AI with the expected JSON shape to generate test inputs. This allows your tests to be simple, concise, [pure functions](https://docs.scala-lang.org/scala3/book/fp-pure-functions.html) using the bare minimum to verify your deserialization strategy. That will very likely be the problem. If it isn't, then you can move on to other possible causes. Think about what those might be. For example, maybe HTTP headers are causing Spring Boot not to activate JSON deserialization. No matter what, lazy load complexity as you go. Spring Boot offers [HTTP mocking](https://spring.io/guides/gs/testing-web). If you feel like you need a network and real HTTP, upgrade your test to use Testcontainers. If all else fails, then you can test against the real API, but I would personally prefer to improve observability to look for the problem. Then when you have a suspect in the case, write a test for your theory. By the way, these ideas go beyond Spring Boot and even Java. They apply to any problem in any stack. Identify the core concern you want to test and do it with as little ceremony as possible. And please forget about [code coverage](https://www.vidyasource.com/blog/code-coverage-is-killing-you/). Unless you have a boss who doesn't know any better. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "How Sports Busts Common Myths About Software Engineering Teams" description: "Learn how sports history reveals why great engineering teams need more than star talent - they need systems, culture, and leadership that elevate everyone's performance." canonical: https://www.vidyasource.com/blog/how-sports-busts-common-myth-software-engineering-teams/ type: article published: 2025-05-21 author: "Neil Chaudhuri" tags: ["Project Management", "Agile"] image: https://www.vidyasource.com/img/blog/football.jpg --- # How Sports Busts Common Myths About Software Engineering Teams > Learn how sports history reveals why great engineering teams need more than star talent - they need systems, culture, and leadership that elevate everyone's performance. - Canonical page: https://www.vidyasource.com/blog/how-sports-busts-common-myth-software-engineering-teams/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/how-sports-busts-common-myth-software-engineering-teams.json - Published: 2025-05-21 - Author: Neil Chaudhuri - Topics: Project Management, Agile With tech layoffs continuing to sap morale and self-proclaimed experts on engineering productivity offering unlimited takes, one take in particular struck me: > Mediocre teams build mediocre products. Period. Every hire should raise the bar—if they don’t, don’t hire them. The harsh truth: A small team of A-players will crush a bloated team of B and C players every time. Talent density beats headcount. Mediocrity drags everyone down. I am all for hiring great people and showing them how much you value them by providing excellent pay, training for growth, paid leave, and other truly beneficial means of compensation rather than pizza, which says a lot since I really love pizza. It's hard to argue against the proposition that your engineering success is a function of how many "A-players" are on your team. But the history of sports, past and recent, shows it isn't that simple. Even worse, that sentiment is harmful. So who are "the best"? Starting with the 1992 Dream Team, the United States assembled the best NBA players, the best of the best, to represent America at the Olympics and other world basketball competitions. The Dream Team won handily, but in the decades that followed, it didn't take long for other national teams who were far less talented on paper to [beat Team USA](https://sports.yahoo.com/6-games-team-usa-mens-214728276.html). Meanwhile, sports fans around the world over decades have also watched many other teams attempt to "buy a championship"—*i.e.* to spend a lot of money on high-priced free agent "A-players"—only to fail so miserably so often that it's a cliché. If team building is as simple as hiring the best individuals, then why do so many sports teams who try fail? It's the implicit notion that "the best" are the best in a vacuum. It is even worse than "harsh" to sort your people into "A-players" and "B and C players." It's self-serving. It is far too easy for management to absolve themselves of responsibility for poor performers. We see it all the time because good leadership is hard, and it's easier to blame other people when things go sideways. In the NFL last season, two quarterbacks cast off by the teams who drafted them, Jared Goff and Sam Darnold, [competed for the best record in the NFC](https://www.mlive.com/lions/2025/01/jared-goff-vs-sam-darnold-once-castoffs-now-an-elite-qb-matchup.html) with their new teams while the teams that cast them off (especially Darnold's) are faring not so great. I know you've had it with my sports metaphors by now, so I want to make clear that management should lead by investing in systems and structures that create a culture that allows people to enjoy flow state to deliver their best as a cohesive unit, a real team, rather than an assemblage of individuals. And when they do, compensate them. Let's refrain from stigmatizing individuals. Instead let's figure out how managers can become leaders who will meet the moment. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "No One Likes Cognitive Load: But Whose Cognitive Load?" description: "Minimizing cognitive load is more important than maximizing clean code, but cognitive load is subjective. Learn how to establish team standards that enhance productivity and maintainability." canonical: https://www.vidyasource.com/blog/no-one-likes-cognitive-load-but-whose-cognitive-load/ type: article published: 2025-05-18 author: "Neil Chaudhuri" tags: ["Programming", "Functional Programming", "Software Engineering", "Agile"] image: https://www.vidyasource.com/img/blog/conditional.png --- # No One Likes Cognitive Load: But Whose Cognitive Load? > Minimizing cognitive load is more important than maximizing clean code, but cognitive load is subjective. Learn how to establish team standards that enhance productivity and maintainability. - Canonical page: https://www.vidyasource.com/blog/no-one-likes-cognitive-load-but-whose-cognitive-load/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/no-one-likes-cognitive-load-but-whose-cognitive-load.json - Published: 2025-05-18 - Author: Neil Chaudhuri - Topics: Programming, Functional Programming, Software Engineering, Agile Addy Osmani works at Google as an Engineering Leader on Chrome and is a renowned author and speaker. He posted [advice on LinkedIn](https://www.linkedin.com/feed/update/urn:li:activity:7275880173600743424/) that I would summarize this way: The impulse to write "clean code" usually add complexity that increases cognitive load and only makes things worse, so your goal should always be reducing cognitive load rather than writing clean code. I have some experience with this. I once worked on a code base that had the cleanest, most elegant code I have ever seen, and I couldn't make sense of it at all. Interfaces with one method and one- or two-line functions strewn throughout. They were very proud of it too. You would find comments like "// Applying the Strategy Pattern here." It was like they wrote code for a conference talk instead of a customer. So I agree completely with Addy in principle. It is more important that code is easy to reason about than that it's "clean," but there is one big problem: Different teams find different code easy to reason about. Take this example from the [GitHub repo](https://github.com/zakirullin/cognitive-load) Addy cites: ![Author's example of complex conditionals and a better alternative](https://www.vidyasource.com/img/blog/conditional.png) This cosmetic change doesn't ease the cognitive burden or fix the real problem, which is that conditionals are inherently complex. I'd say the solution that fixes the problem is to introduce a [state machine](https://stately.ai/docs/state-machines-and-statecharts) to eliminate the conditionals and to guarantee that all states are accounted for. State machines are clear, explicit about intent, and shift feedback left by informing you of errors at compile time, but of course someone else might say they find state machines complicated. This goes for lots of coding concepts. I also find that static types and immutability reduce cognitive load, but a JavaScript or Python dev might not be so sure. The GitHub page warns of modules that are too small or too big, but how many lines of code is the sweet spot in between? I think the right strategy is to reduce cognitive load but in a manner specific to your team. How? Here are some ideas: * Use your feedback mechanism, whether it's pair programming or code reviews in a PR or even AI, to have conversations about what makes sense. Build a team consensus. Conversations can be informal too. * Once you have some team heuristics on what your code should look like, enforce them where possible with Architectural Fitness Functions using tools like ArchUnit or PyTestArch. Or AI. * Be specific in your AI prompts or better yet in AI config files (*e.g.* `.cursorrules`,`.junie/guidelines.md`, `.goosehints`, *etc*) about what the code it generates should look like to conform to team norms. So don't worry about making your code clever or "clean." Remember that your primary responsibility is to your team, your leadership, your organization, and your customer—not to yourself for coming up with stimulating intellectual challenges you'll enjoy. Figure out what works for your team to make code as simple to read, understand, and maintain as possible. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Programming Without Ifs: Better Alternatives to Conditional Logic" description: "Discover how to write more maintainable code by replacing conditional logic with practical alternatives that reduce complexity and improve your software.\"" canonical: https://www.vidyasource.com/blog/programming-without-ifs-better-alternatives-to-conditional-logic/ type: article published: 2025-05-16 author: "Neil Chaudhuri" tags: ["Scrum", "Kanban", "Selenium", "Programming", "Software Engineering", "Architecture", "Testing"] image: https://www.vidyasource.com/img/blog/programming.jpg --- # Programming Without Ifs: Better Alternatives to Conditional Logic > Discover how to write more maintainable code by replacing conditional logic with practical alternatives that reduce complexity and improve your software." - Canonical page: https://www.vidyasource.com/blog/programming-without-ifs-better-alternatives-to-conditional-logic/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/programming-without-ifs-better-alternatives-to-conditional-logic.json - Published: 2025-05-16 - Author: Neil Chaudhuri - Topics: Scrum, Kanban, Selenium, Programming, Software Engineering, Architecture, Testing I have always stood on a programming soap box that conditionals and booleans are insidious. They are like the second thing we learn on the first day of class and are so straightforward, but before you know it, they've added so much sneaky complexity to our code in the real world! Here are a few alternatives to conditionals and booleans agnostic of programming language: ## State machines Use state machines to model your workflow rather than conditionals to test repeatedly where you are in a workflow. For example, XState in JavaScript/TypeScript and Temporal in many languages both provide state machines. ## Content negotiation If your API needs to respond with either JSON or XML based on an HTTP `Accept` header, configure a post-processor that serializes the output into the right rendering once you've represented it as an object in memory. For example, Spring Boot developers writing in Java and Kotlin do this with `MessageConverter`s by default. ## Maps Use a map data structure rather than conditional logic to fetch data. Every language has maps. ## Guards, filters, and interceptors Different programming languages and frameworks use these terms in different ways, but broadly speaking, these are all ways inspired by Aspect-Oriented Programming to decouple the primary business logic of an HTTP API endpoint from cross-cutting decisions that have to be made in advance --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Waterfall's \"Scope Creep\" is Agile's Revelation" description: "Discover why \"scope creep\" isn't a problem but a valuable insight. Learn how agile methodology embraces change to build better products while Waterfall's rigid planning leads to missed opportunities." canonical: https://www.vidyasource.com/blog/waterfall-scope-creep-is-agile-revelation/ type: article published: 2025-05-10 author: "Neil Chaudhuri" tags: ["DevOps", "Project Management", "Agile"] image: https://www.vidyasource.com/img/blog/scope-creep.jpg --- # Waterfall's "Scope Creep" is Agile's Revelation > Discover why "scope creep" isn't a problem but a valuable insight. Learn how agile methodology embraces change to build better products while Waterfall's rigid planning leads to missed opportunities. - Canonical page: https://www.vidyasource.com/blog/waterfall-scope-creep-is-agile-revelation/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/waterfall-scope-creep-is-agile-revelation.json - Published: 2025-05-10 - Author: Neil Chaudhuri - Topics: DevOps, Project Management, Agile I saw someone post this on LinkedIn: "How do you deal with scope creep in an agile software development project?" I find this question so aggravating in 2025 that I just had to write a quick post about it. When you set out to build a new product, you start with good faith assumptions that are unfortunately almost certainly wrong about what your customers want and what it takes to deliver that. As with any creative work, it's inevitable. Waterfall denies that reality and, despite unanimous evidence to the contrary, believes months or even years of methodical planning will avoid it. A reality so gross that waterfall labels it the same way we label the weird dude at the bar who won't take a hint: creep. Instead, agile (and lean) accepts that reality. It isn't gross. It's just that none of us are Steve Jobs. We will get things wrong. It's OK. Even Tony Stark [gets things wrong](https://www.youtube.com/watch?v=1w8sHzLtWAY). Agile product development offers ways to minimize the fallout—namely experimenting fast to discover what you got wrong fast so you can pivot off the feedback fast and deliver the *right* thing. Fast. So what waterfall calls "creep" is actually a welcome revelation of what the right path should have been all along if we had Jobs-ian clairvoyance at the jump, and agile helps us minimize the cost of our bad initial assumptions. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Fibonacci Theater: Adding Mathematical Mystique to Scrum Estimation" description: "Story points are the fundamental metric in Scrum, but using the Fibonacci Sequence to estimate them is more gimmick than substance." canonical: https://www.vidyasource.com/blog/fibonacci-theater-adding-mathematical-mystique-to-scrum-estimation/ type: article published: 2025-01-18 author: "Neil Chaudhuri" tags: ["Agile", "Scrum", "Project Management", "Software Engineering", "Testing", "Continuous Integration"] image: https://www.vidyasource.com/img/blog/fibonacci.jpg --- # Fibonacci Theater: Adding Mathematical Mystique to Scrum Estimation > Story points are the fundamental metric in Scrum, but using the Fibonacci Sequence to estimate them is more gimmick than substance. - Canonical page: https://www.vidyasource.com/blog/fibonacci-theater-adding-mathematical-mystique-to-scrum-estimation/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/fibonacci-theater-adding-mathematical-mystique-to-scrum-estimation.json - Published: 2025-01-18 - Author: Neil Chaudhuri - Topics: Agile, Scrum, Project Management, Software Engineering, Testing, Continuous Integration I saw this question on LinkedIn and hear it a lot, so I figured I'd write a post for anyone else wondering the same thing. ![Scrum Masters & Agile Coaches: Why do we use Fibonacci Sequence 1,2,3,5,8 etc. for estimation instead of just Natural numbers like 1,2,3,4 etc?](https://www.vidyasource.com/img/blog/fibonacci-question.jpeg) Story points are the fundamental metric in Scrum, which through sustained, effective marketing has become the default agile methodology. Story points have been a source of confusion for decades because it's easy to mistake what they represent. Story points are not hours or a measure of anything particularly concrete but rather a fuzzy representation of the size, scope, complexity, and value of a software feature relative to one another. Assigning one feature 2 points is marginally useful, but knowing that feature is a lot easier than that 8 over there is much more useful. Another fundamental notion in Scrum, and really agile writ large, is uncertainty, which is a profound element of software engineering. We need to be intentional about recognizing what we don't know yet. Story point estimates grow less and less useful with more and more uncertainty. Because the Fibonacci Sequence grows exponentially rather than linearly, Fibonacci numbers are a clever (if gimmicky) symbol of an idea—the idea that even the slightest increase in uncertainty yields a significant increase in our story point estimate. It is really important to remember that the inclusion of fancy-seeming math-iness like Fibonacci in Scrum is not some kind of research-derived correlation between engineering effort and productivity or anything remotely provocative like that. It's just symbolic of the consequences of uncertainty. The confusion around Fibonacci's relevance to Scrum is one example of why I believe story points are the key reason Scrum fails so often, but that is a story for another day. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Vidya Talks Modernizing Tech Hiring in the Federal Government at the White House" description: "Vidya spoke to tech recruiters about how the federal government can hire and retain the best tech talent." canonical: https://www.vidyasource.com/blog/vidya-talks-modernizing-tech-hiring-federal-government-white-house/ type: article published: 2024-10-12 author: "Neil Chaudhuri" tags: ["Partners", "Diversity", "Programming", "Software Engineering", "Agile", "Government"] image: https://www.vidyasource.com/img/blog/tech-hiring.webp --- # Vidya Talks Modernizing Tech Hiring in the Federal Government at the White House > Vidya spoke to tech recruiters about how the federal government can hire and retain the best tech talent. - Canonical page: https://www.vidyasource.com/blog/vidya-talks-modernizing-tech-hiring-federal-government-white-house/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/vidya-talks-modernizing-tech-hiring-federal-government-white-house.json - Published: 2024-10-12 - Author: Neil Chaudhuri - Topics: Partners, Diversity, Programming, Software Engineering, Agile, Government About 20 years ago or so, commercial businesses started to figure out they were building software all wrong. Waterfall software development, with its phased gates and handoffs and large chunks and wasted time, is not conducive to the experimentation and rapid feedback that creative work like software engineering demands. This led to a transition, difficult and controversial as it's been, to agile product development. Government doesn't have the advantages the private sector has, so fundamental change like that takes far longer in the public sector. Still, the right leadership can move things forward. It started with the creation of [United States Digital Service (USDS)](https://www.usds.gov/) and [18F](https://18f.gsa.gov/), which rotate elite talent from industry into government to provide their leading edge expertise to government agencies. Ten years ago, I was part of a team working with USDS and 18F to teach federal procurement officials how to write better contracts to motivate better behavior from vendors and ultimately to deliver better software. That curriculum evolved into [DITAP](https://techfarhub.usds.gov/get-started/ditap/). Now in 2024 after decades of outsourcing software product development to private vendors, the federal government wants to bring more subject matter expertise in house. There is a significant effort underway to improve government hiring practices by applying the best lessons from industry, which to be fair has not mastered hiring in its own right, to staff agencies with the [talent that tech laid off](https://www.vidyasource.com/blog/the-mckinsey-developer-productivity-analysis-is-a-ruse/) after the pandemic. To that end, I joined another team a few weeks ago on Technology Day at the White House—yes, *that* White House—to give a talk to a cohort of federal recruiters. I'm not going to lie. I was nervous but excited for the opportunity. ![Neil Chaudhuri in suit and sunglasses.](https://www.vidyasource.com/img/blog/tech-hiring-shades.webp) The talk was called *Tech Landscape & Government Realities: Modernizing Government Hiring.* ![Tech Landscape & Government Realities: Modernizing Government Hiring cover slide](https://www.vidyasource.com/img/blog/tech-hiring.png) I began with the bad news and the good news for federal recruiters. The bad news is that there is a lot of elite tech talent that looks at Big Tech companies like Uber, Meta, Spotify, Google, *etc.* the same way I look at Major League Baseball, a lifelong dream job, and it is all but impossible to change their minds. The good news is that there remains plenty other elite talent available now who recognizes there are disabled people, elderly people, veterans suffering the consequences of war, and countless others who need help, talent who wants to make it possible for them to live their lives with dignity. Meanwhile, government agencies are remaking their hiring processes and organizational structures to identify job titles and define career paths better aligned to what candidates are used to in private industry. ![Neil Chaudhuri presenting examples of government leadership on a slide to the audience.](https://www.vidyasource.com/img/blog/tech-hiring-leadership.webp) I then went on to describe what makes [flow state](https://leaddev.com/culture-engagement-motivation/why-flow-matters-more-passion) the foundation for a good work culture and how flow state hews closely to the principles of agile development. Finally, I walked through multiple examples of popular software product categories that industry and government have both built, from websites to machine learning pipelines to AI, and the kinds of job roles best suited to deliver them. ![Neil Chaudhuri presenting flow state on a slide to the audience.](https://www.vidyasource.com/img/blog/tech-hiring-flow-state.webp) It can be hard to know how a talk went after it's over and the polite applause dies down. It was gratifying when many of the recruiters in the audience came up to me afterwards to tell me they enjoyed it and learned a lot from it. One even told me that he had similar ideas himself but had lacked the vocabulary to express them to his leadership until seeing my presentation and that he was excited to make big, forward-looking changes to his agency's hiring process. A colleague informed me that students had told him they "loved the presentation." I am just glad that the audience was able to glean something useful to implement at their respective agencies from the mix of information, jokes, and pop culture references that typically define my talks. It was an honor to present at the White House, and I hope I have the opportunity to continue to help these recruiters and others down the line modernize hiring in the federal space and ultimately help our government deliver the best digital services of any government in the world. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Lessons from The Bear for Tech Leaders" description: "Discover unexpected leadership insights from 'The Bear.' Learn how tech leaders can improve candidate experiences and foster employee success. Valuable lessons from an Emmy-winning series." canonical: https://www.vidyasource.com/blog/lessons-from-the-bear-for-tech-leaders/ type: article published: 2024-08-12 author: "Neil Chaudhuri" tags: ["Leadership", "Project Management", "Agile"] image: https://www.vidyasource.com/img/blog/bear-tina-mikey.jpg --- # Lessons from The Bear for Tech Leaders > Discover unexpected leadership insights from 'The Bear.' Learn how tech leaders can improve candidate experiences and foster employee success. Valuable lessons from an Emmy-winning series. - Canonical page: https://www.vidyasource.com/blog/lessons-from-the-bear-for-tech-leaders/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/lessons-from-the-bear-for-tech-leaders.json - Published: 2024-08-12 - Author: Neil Chaudhuri - Topics: Leadership, Project Management, Agile I was late to the party, but I now understand why The Bear is an Emmy award winning TV show. The story is fascinating, and the ensemble cast is among the best casts ever. It's a show about a restaurant, but I watched an outstanding episode recently that offers surprising lessons for tech leaders. ## A quick summary of The Bear If you've never seen it, The Bear is a dramatic comedy about Carmy Berzatto, a young chef from a working class family in Chicago who leaves a difficult past behind to train with the greatest chefs in the world and learn the high pressure world of fine dining. Carmy is gifted, intense, sensitive, and terrified of failure, and these traits come into focus in tragedy when he must return home because his beloved older brother Mikey, a pillar of the community and owner of a popular sandwich shop called The Beef, has ended his own life. To Carmy's surprise because Mikey never let him work at The Beef, Mikey left the restaurant to him, so he sets about bringing his experience with high-end, fastidious kitchens to a decidedly chaotic kitchen. I will leave it there to avoid spoilers, but The Bear is about much more than Carmy. Over the course of three seasons so far, the show introduces us to an array of fascinating characters with their own compelling stories. One of them is Tina, a line cook fiercely loyal to Mikey who bristles with hostility at the changes Carmy imposes to raise the quality of The Beef. In the best episode of Season 3, Episode 6 called "Napkins," we learn exactly why Tina is so loyal to Mikey, and her story offers lessons that tech leaders would do well to heed. ## Lesson 1: Treat job candidates with respect. "Napkins" is a flashback episode that shows us how Tina met Mikey and became a cook at The Beef. After 15 years at a company doing payroll, they lay her off, so at age 46 Tina faces a world that has transformed since the last time she was in the job market. It's all about LinkedIn now instead of professional rapport and well-formatted resumes. Tina's optimism and confidence melt into frustration and disappointment as one Gen Z HR kid after another dismisses her with a perfunctory "We'll let you know." Meanwhile, her husband makes little as a doorman as his boss casually dangles the possibility of a promotion that never materializes, and they have a son. The struggle of it all makes Tina feel worthless. As tech leaders, we make choices that define the culture of our organizations. While it may be easier to exert draconian command and control that promotes fear of reprisal, it's healthier and better in the long run to promote psychological safety, collaboration, and respect. That will naturally extend to people who express a desire to join your organization. For reasons we can discuss another day, tech seems to have more skepticism at best and contempt at worst for job candidates than any other industry. Instead, honor their interest in joining your team by respecting their time and acknowledging that they are three-dimensional human beings whose livelihoods may very well be in your hands. ## Lesson 2: Your people don't need "passion" to excel in tech. After a particularly hard day where even the bus adds insult to injury by running 29 minutes late, Tina wanders around for a place to wait and sees The Beef. She walks in, observes the chaos of lunchtime to-go orders, and receives her first pleasant surprise in a long time—a free sandwich because no customer is around to claim it. Tina finds a quiet table in the back, takes a bite, and finally unleashes the tears she has contained for weeks. Mikey notices her and sits with her. What follows is a compelling conversation. ![Tina and Mikey meeting for the first time and having a real heart to heart conversation](https://www.vidyasource.com/img/blog/bear-tina-mikey.jpg) Tina and Mikey discover they have a lot in common. It turns out Mikey is dealing with his own problems at the restaurant like an unfixable toilet and insufficient staff (Put a pin in that!) and also opens up, remarkably given that he just met Tina, about a heartbreaking lesson he internalized on a field trip as a kid—that he will always struggle because he has no innate gifts. Tina returns the favor by sharing her own struggles. By the end of their conversation, Mikey offers Tina a job at The Beef to bring a merciful end to her months-long struggle, and these few minutes forge a bond between them that will endure even beyond Mikey's tragic demise. Something Tina tells Mikey really resonates with me: > I don’t need to be inspired. I don’t need to be impassioned. I don’t need to make magic. I don’t need to save the world, you know? I just wanna feed my kid. Every other week on Tech Twitter some Blue Check CEO, out of misguided sincerity or a selfish desire to start an argument for engagement and easy money, declares that no one can succeed in tech absent daily open source contributions, multiple side projects, a high score on HackerRank, eagerness to work weekends to make their bosses rich, and other emblems of "passion" for writing code. It's provocative, but nothing could be further from the truth. People who work in your organization only because they want to afford to "feed their kid" will bring just as much value as people with passion for the work. Let's talk about two major problems with the requirement for passion. Activities like side projects demand time, which is a rare luxury for parents and especially mothers as women disproportionately bear the burden of childcare in America. Time can be precious for many other reasons as well, and erecting these artificial barriers to success in tech is unfair to everyone but the most privileged. Beyond gatekeeping, decades of research bears out that it is not the job of engineers to come into work with passion but instead our job as tech leaders to create [flow state](https://leaddev.com/culture-engagement-motivation/why-flow-matters-more-passion) so they can succeed. It is on us to provide clarity of purpose, achievable goals, psychological safety, ample feedback, fair compensation, and empowerment to make decisions. Offloading our responsibility to create flow state onto devs to walk in with passion is a self-serving dereliction of duty. ## Reflections Tech leaders have a lot to learn to make our industry as vibrant as it can be, and we need to learn our lessons wherever we find them. The Bear is an award-winning TV show about a restaurant, an unlikely source for lessons for the tech industry, but Season 3 Episode 6 "Napkins" teaches us how we can serve job seekers on the outside by showing empathy and respect and how we can serve our own people on the inside by creating flow state so they can do their best work regardless how much passion they have for tech on their own time. Let's continue to to be on the lookout for opportunities to make ourselves better for our people, our customers, and ourselves. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Modernizing Online Passport Renewal: Vidya's Success at the State Department" description: "Vidya helped develop the architecture for Online Passport Renewal, which is now live in beta at the State Department." canonical: https://www.vidyasource.com/blog/modernizing-online-passport-renewal-vidya-success-at-the-state-department/ type: article published: 2024-07-29 author: "Neil Chaudhuri" tags: ["Partners", "Government", "Project Management", "Agile", "Programming", "Testing", "Continuous Delivery", "Open Source", "Accessibility", "APIs", "AI"] image: https://www.vidyasource.com/img/blog/passport.jpg --- # Modernizing Online Passport Renewal: Vidya's Success at the State Department > Vidya helped develop the architecture for Online Passport Renewal, which is now live in beta at the State Department. - Canonical page: https://www.vidyasource.com/blog/modernizing-online-passport-renewal-vidya-success-at-the-state-department/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/modernizing-online-passport-renewal-vidya-success-at-the-state-department.json - Published: 2024-07-29 - Author: Neil Chaudhuri - Topics: Partners, Government, Project Management, Agile, Programming, Testing, Continuous Delivery, Open Source, Accessibility, APIs, AI In an era where digital transformation in government is paramount to deliver the most value to customers in America and around the world, Vidya has played a major role in modernizing crucial government services. Our collaboration with the United States Department of State's Bureau of Consular Affairs has provided new opportunities for efficiency, user experience, and technological innovation. ## Transforming essential services The Bureau of Consular Affairs provides essential services for customers worldwide. Here are just a few: - Issuing Certificates of Birth Abroad (CRBA) for children born to American parents overseas - Processing visas for international visitors - Providing passports for U.S. citizens embarking on global adventures To provide these services over the years, the State Department has constructed an elaborate software and data architecture to automate as much as possible. As all large organizations inside and outside government understand, it is critical to invest in modernization to improve user interfaces, performance, and security and ultimately make the user experience as efficient and intuitive as possible for both State employees and customers. ## The modernization challenge In 2019, Vidya joined the most recent modernization effort as software architects. Anyone who has worked on large-scale modernizations knows they aren't just about updating a few lines of code. Modernization is about reimagining entire systems while ensuring uninterrupted service delivery to customers who depend on it. For those less familiar, there are techniques like [feature flags](https://martinfowler.com/articles/feature-toggles.html), [blue/green deployments](https://docs.aws.amazon.com/whitepapers/latest/overview-deployment-options/bluegreen-deployments.html), and [canary deployments](https://cloud.google.com/deploy/docs/deployment-strategies/canary). You probably know Martin Fowler as one of the signatories of the Agile Manifesto, and it seems like half his website is dedicated to modernization with discussions of [seams](https://www.martinfowler.com/articles/uncovering-mainframe-seams.html) and [integration](https://martinfowler.com/articles/cant-buy-integration.html) among many other related topics. On top of technology, it is essential to create a culture of innovation with disciplines like product management, continuous deployment, and organizational psychology. For all these reasons, modernization is one of the most challenging feats in our field. ## Pioneering success: eCRBA and Online Passport Renewal Our first major triumph was the launch of eCRBA, streamlining the process for American parents to obtain birth certificates for their children born overseas. This service has been a game-changer, offering unprecedented convenience and efficiency. The crown jewel of our modernization efforts, Online Passport Renewal (OPR), is now live. The most recent analysis of OPR revealed some welcome accomplishments: - Processing thousands of passports daily - Boasting a remarkable 97% customer satisfaction rate and 80% increased trust in government - Generating neutral to positive social media sentiment Don't just take our word for it! Sean Hollister at The Verge had a [good experience with OPR](https://www.theverge.com/2024/7/3/24190366/us-passport-online-renewal-beta-fast). Forbes Magazine [highlighted its innovative approach](https://www.forbes.com/sites/maryroeloffs/2024/06/12/online-passport-renewal-launch-how-does-it-work-am-i-eligible-united-states-/) as well. ## The road ahead While we at Vidya celebrate these milestones, we're far from done. There is much more work to do: - Scaling up OPR to handle full production workloads - Modernizing additional State Department services - Continuously refining and optimizing our solutions There may be even room for AI here, but as urgent as it is to promote safety in AI in general, it's infinitely more so for government systems. We will need to explore how to maximize safety and explainability while minimizing hallucinations and toxicity to protect our customers because AI vendors themselves will not. Whether it's AI or any number of other innovations we have in mind, we look forward to working with the State Department to see how we can best leverage their capabilities for our mission. ## Your partner in government modernization At Vidya, we don't just build software. We revolutionize processes. We enhance user experiences. We even bring a touch of humor to even the most complex projects. Our success with the State Department is just the beginning. [Join us](https://www.vidyasource.com/contact/) as we continue to push the boundaries of what's possible in government modernization. With Vidya, the future of streamlined, efficient government services is not just a possibility. It's a reality we're creating every day. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Vidya Awarded a GSA MAS Contract" description: "Vidya is proud that GSA has awarded us a Multiple Award Schedule contract for providing IT Professional Services to the government." canonical: https://www.vidyasource.com/blog/vidya-awarded-gsa-mas-contract/ type: article published: 2024-05-30 author: "Neil Chaudhuri" tags: ["Partners", "Diversity", "Programming", "Software Engineering", "Agile", "Lean", "Government"] image: https://www.vidyasource.com/img/blog/gsa-landscape.jpg --- # Vidya Awarded a GSA MAS Contract > Vidya is proud that GSA has awarded us a Multiple Award Schedule contract for providing IT Professional Services to the government. - Canonical page: https://www.vidyasource.com/blog/vidya-awarded-gsa-mas-contract/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/vidya-awarded-gsa-mas-contract.json - Published: 2024-05-30 - Author: Neil Chaudhuri - Topics: Partners, Diversity, Programming, Software Engineering, Agile, Lean, Government I am excited to announce Vidya has been awarded a GSA Multiple Award Schedule (MAS) contract for providing IT Professional Services to the government. Our GSA MAS contract gives us the opportunity to join other contract holders in a marketplace so we can sell our software architecture, software engineering, and software consulting services directly to the federal government. Government agencies can trust that Vidya has been vetted for a history of delivering high-quality software to a wide array of government and commercial clients using exciting technology while promoting lean principles, agile software development, and diversity in software engineering. Vidya can enjoy direct access to government customers who need our services with faster award times. It's a win-win! I am confident that Vidya has even more to offer government customers particularly in the area of architecture and modernization using powerful techniques and technologies proven in industry, and this GSA MAS contract award confirms that government shares my confidence. For those well versed in GSA Schedules, Vidya's Special Item Number (SIN) is [54151S](https://www.gsa.gov/technology/it-contract-vehicles-and-purchasing-programs/multiple-award-schedule-it/it-professional-services). We look forward to building new relationships and to delivering our services to even more government agencies. We are also excited about expanding our current offerings by making our courses available online and building new software products that we expect will also be on a GSA Schedule one day. We want to thank GSA for recognizing Vidya's past success and future potential, and we cannot wait to do business with you. [Let's talk](https://www.vidyasource.com/contact/). --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Vidya is on the Pod" description: "Vidya President Neil Chaudhuri appeared on the Guidance Counselor 2.0 Podcast with Taylor Desseyn." canonical: https://www.vidyasource.com/blog/vidya-on-the-pod/ type: article published: 2023-10-17 author: "Neil Chaudhuri" tags: ["Agile", "Big Tech", "Software Engineering"] image: https://www.vidyasource.com/img/blog/guidance-counselor-pod.png --- # Vidya is on the Pod > Vidya President Neil Chaudhuri appeared on the Guidance Counselor 2.0 Podcast with Taylor Desseyn. - Canonical page: https://www.vidyasource.com/blog/vidya-on-the-pod/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/vidya-on-the-pod.json - Published: 2023-10-17 - Author: Neil Chaudhuri - Topics: Agile, Big Tech, Software Engineering I had the opportunity to appear on the Guidance Counselor 2.0 Podcast with [Taylor Desseyn](https://twitter.com/tdesseyn), one of the best tech recruiters in the country and an expert at helping recruiters and engineers understand each other. Follow Taylor to learn more about how to get the right job in tech for you. Please check out [my conversation with Taylor](https://www.linkedin.com/video/live/urn:li:ugcPost:7104832496722173953/). We discussed [what makes the McKinsey developer productivity analysis so terrible](https://www.vidyasource.com/blog/the-mckinsey-developer-productivity-analysis-is-a-ruse/), how developers can achieve [flow state](https://leaddev.com/culture-engagement-motivation/why-flow-matters-more-passion), the importance of adapting your remote work policies to the needs and culture of your organization, and a lot more. I hope you enjoy it! Feel free to [let us know what you think](https://www.vidyasource.com/contact/). --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "The McKinsey Developer Productivity Analysis Is a Ruse" description: "The McKinsey developer productivity analysis is worse than wrong. It's bad faith. Your organization can do so much better." canonical: https://www.vidyasource.com/blog/the-mckinsey-developer-productivity-analysis-is-a-ruse/ type: article published: 2023-10-09 author: "Neil Chaudhuri" tags: ["Leadership", "AI", "Government", "Programming", "Testing", "Security", "Cloud Computing", "Software Engineering", "Architecture"] image: https://www.vidyasource.com/img/blog/mckinsey.jpg --- # The McKinsey Developer Productivity Analysis Is a Ruse > The McKinsey developer productivity analysis is worse than wrong. It's bad faith. Your organization can do so much better. - Canonical page: https://www.vidyasource.com/blog/the-mckinsey-developer-productivity-analysis-is-a-ruse/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/the-mckinsey-developer-productivity-analysis-is-a-ruse.json - Published: 2023-10-09 - Author: Neil Chaudhuri - Topics: Leadership, AI, Government, Programming, Testing, Security, Cloud Computing, Software Engineering, Architecture There is a popular debate on Tech Twitter that rears its head every few weeks but has become particularly salient recently. No, not the one about whether HTML is a programming language. Or the one about whether Tailwind CSS is good. Or the one about whether microservices are a disaster. Or the one about whether you need to hustle and grind to have a career in tech. Or even the one about whether you need a computer science degree to excel in tech. (We sure do have a lot of dumb whack-a-mole arguments in tech, don't we?) It's the one about whether, and how, you can measure the productivity of individual software developers in an organization. The industry is focused on the idea lately since the job market in tech has faced upheaval coming out of the pandemic and since famous CEOs have brazenly made it clear they think very little of the tech workforce—from Elon Musk's mission to fire anyone who isn't "[extremely hardcore](https://www.theregister.com/2022/11/16/musk_twitter_ultimatum/)" to Mark Zuckerberg's "[year of efficiency](https://www.reuters.com/technology/meta-lays-off-tech-teams-battering-employee-morale-2023-04-19/)." It turns out this isn't a debate at all: _You cannot measure individual productivity._ As we will see shortly, research has settled the question, but that hasn't stopped people from [trying to make fetch happen](https://www.youtube.com/watch?v=Pubd-spHN-0). McKinsey recently released a report called *[Yes, you can measure software developer productivity](https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/yes-you-can-measure-software-developer-productivity)*. As the title suggests, the authors affirmatively declare that you *can* measure individual productivity, and they offer a framework for doing so. McKinsey is wrong, but it's worse than that. McKinsey is motivated now more than ever to offer tech managers this fiction, and it's all just a ruse to get paid for telling CEOs what they want to hear. ## Who is McKinsey anyway? McKinsey is a management consultant company. In theory, they help managers become better managers. McKinsey is like [The Bobs in Office Space](https://www.youtube.com/watch?v=j_1lIFRdnhA) with the elite pedigree of ["the kids" in Succession](https://succession.fandom.com/wiki/Roy_family). Management consulting isn't a bad idea. In fact, it's a great idea! We have all known managers who...aren't awesome. They can use help. Luminaries like Eliyahu Goldratt, Susan David, Peter Drucker, Amy Edmonson, Daniel Goldman, Adam Grant, and many others have made significant contributions to management consulting and organizational psychology. The problem is McKinsey [gets a lot wrong](https://x.com/TrungTPhan/status/1688583089323438080?s=20) and, even worse, has [sold its "expertise" to the worst clients and themselves have done legally dubious things](https://fortune.com/2023/06/21/mckinsey-hiring-ethics-salary/). Now their ethical and moral lapses, and even their long track record of really bad predictions, don't necessarily mean McKinsey's analysis of software development productivity is wrong, but they do provide context for how McKinsey does business and why they might find it valuable to weigh in on the issue. ## What's in it for McKinsey ? With all the problems in the world, why would McKinsey spend time analyzing the feasibility of measuring individual software development productivity? I am a retail investor, and one thing I have learned is Big Tech does not optimize for profit *per se* but for share price. You may know Big Tech stocks took a big hit coming out of the pandemic for a lot of reasons, and there are two things Wall Street wants to see from companies in trouble that they want to see succeed: layoffs and buybacks (companies using their cash on hand to buy their own stock). So predictably and sadly, Big Tech recently engaged in mass layoffs. In waves! Lest you think it was because of a lack of resources to pay people, [they had the money to execute buybacks as well](https://finance.yahoo.com/news/tech-giants-embrace-stock-buybacks-120012389.html). The Street loved the combination as it always does. Big Tech stock went up once again. The good news for software engineers is The Street now wants to see big investments in AI, so Big Tech is slowly hiring again—often the same people they laid off. That tells you it's not about developer performance or the numbers on a balance sheet. Just stock price. So what does all this have to do with McKinsey? McKinsey is a firm with no ethical compunctions about contributing to mass layoffs, and they know there are deep pockets willing to pay them large consulting fees for a framework that makes layoffs look like the result of rigorous academic analysis rather than a cynical ploy for the next shareholder call. It's a ruse. ## What does McKinsey get wrong about measuring the productivity of individual developers? Let's assume for the moment McKinsey did their analysis earnestly in good faith. Then McKinsey does not understand how software development works in the real world because they're consultants not engineers. I recommend you check out [Daniel Terhorst-North's post](https://dannorth.net/mckinsey-review/) also critical of McKinsey's report, including a terrific point on the absence of gender diversity among its authors, for a detailed analysis, but let me add some thoughts. ### McKinsey thinks code is the unit of productivity, so the best engineers churn out the most code It's true that we sell products and products are made of code, but you and I know code is the easy part. So much so that Generative AI can write a lot of it. The hard work is understanding your customers, learning the business domain, reviewing code because not all code is good code, experimenting with new tech, manually verifying accessibility, supply chain security, cleaning up technical debt, integrating with legacy systems, writing tests, automating cloud deployments, mentoring inexperienced engineers, writing documentation, creating a design system, and lots of other things essential to mature software delivery. Put more simply, McKinsey thinks of code like Pez coming out of a dispenser, which is what people who don't build software think building software is like. The reality is it's not about code; it's about _features_ that users will enjoy. Code is only a small part of that. Maybe even the easiest! ### That Outer Loop/Inner Loop Dichotomy is Nonsense McKinsey distinguishes "Outer Loop" activities from "Inner Loop" activities where the latter is productive and the former is waste. As you'd expect, the Inner Loop is about churning out code. They suggest teams should aim for 70% in the Inner Loop. Where does that number come from? Who knows? They made it up like a middle schooler who forgot oral reports are due today. It's also telling they put "Security and Compliance" in the Outer Loop. It's that kind of mentality that causes so many large companies, many of whom are likely McKinsey clients but should know better regardless, to suffer embarrassing breaches that compromise our data. In fact, all the Outer Loop activities are important because they are critical to feature delivery, which is what matters. Users pay for features, not code. ### "Contribution Analysis" and "Talent Capability Score" Have No Basis in Research Based on the book of the same name, the movie [Moneyball](https://en.wikipedia.org/wiki/Moneyball_(film)) is the true story of the Oakland Athletics (aka the A's), a baseball team that lacked the resources to sign the most clearly talented and therefore most expensive players and devised a data-driven approach to assemble cheap, undervalued players into a very good team that could challenge rich teams who could afford the big names. The A's were at the vanguard of the analytics movement that is now commonplace in baseball, and one of the key objectives of this approach is to isolate the value of a particular player from the rest of the team. It's a great idea, but it's really hard and not necessarily always successful. In fact, the most popular modern statistic is Wins Above Replacement (WAR), which seeks to measure how much better one player in isolation is than a "replacement" you can find anywhere, but [there are three variations on WAR](https://www.mlb.com/glossary/advanced-stats/wins-above-replacement) because the three references don't agree on how to calculate it. Measuring the productivity of an individual in the context of a team is all but impossible. It's even harder in software development where so many different, unquantifiable activities go into delivering a feature from talking to users to building design systems to setting up automated deployments to observability and instrumentation to compliance with laws and regulations and on and on. McKinsey's obsession with productivity as lines of code and PRs submitted is reductive and naive. But it gets worse. Let's imagine a developer who is struggling. After all, that happens. It's happened to me. No matter your line of work, we all have Imposter Syndrome and other crises of confidence. If it persists, that isn't the developer's fault. It's the fault of organizational leadership to create the conditions necessary for [flow state](https://leaddev.com/culture-engagement-motivation/why-flow-matters-more-passion). In fact, this leads to the biggest problem I have with the McKinsey report. The authors have the nerve to pay lip service to the [DORA metrics](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance) and the [SPACE framework](https://queue.acm.org/detail.cfm?id=3454124) (more on these shortly), both pioneered by the legendary Dr. Nicole Forsgren and her colleagues on the basis of years of research, only to shove them aside in favor of contrived jargon like "Contribution Analysis" and "Talent Capability Score" and 70% "Inner Loop" activity with the pretense of the same academic rigor. Are you kidding? It would be like if I said "Einstein's great. He's my guy. But I would like to complement his Theory of Special Relativity expressing the equivalence of mass and energy with my own Theory of Artificial Quantum Matrix Superposition that states the nutritional value of [Sour Patch Kids](https://sourpatchkids.com/) is directly proportional to the nutritional value of the fruits they represent. These are two equally valid theories!" This is how you know it's a ruse. McKinsey has no interest in creating a framework to measure developer productivity. Ever the ethically challenged firm, McKinsey has an interest in persuading tech CEOs to pay them lots of money to rationalize layoffs to juice stock prices _under the guise of a research-driven framework_ to measure developer productivity. It's an insult to the tech industry, to our intelligence, and worst of all, to software engineers devoting so much of their time and mental health to organizations that blame them for the failures of management. The good news is we can trust Dr. Forsgren and her teams to show us real, good faith metrics grounded in research that we can use to improve software product delivery. ## Forget McKinsey. Give Yourself SPACE In his seminal book *Lean Startup*, author Eric Ries introduces the concept of the Minimal Viable Product (MVP): "[the] version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort." While tech has taken the MVP to mean an alpha or beta version of a product, Ries means something different—a cheap, lightweight form of the product that helps you validate or reject your assumptions about what customers want before you invest time and money into building it. >>> In the book, Ries makes it clear the product doesn't even have to be tech. He gives an example of a laundry service in India that uses a prop washing machine on a truck as its MVP! Even before you start building and certainly as you continue building, you need to continuously validate your organizational performance (the **_P_** in SPACE) at giving customers and other stakeholders what they want. This is *building the right thing*. You also need to establish a culture of teamwork, psychological safety, flow state, and commitment to quality to promote a delivery pipeline that delivers to users the product they want with the efficiency (the **_E_** in SPACE uniting all the other dimensions) they demand. This is *building the thing right*. With SPACE and DORA metrics, Dr. Forsgren and her teams have given us the blueprint to accomplish both. McKinsey has weaponized their work to promote their own proprietary, lazy nonsense to appease layoff-hungry CEOs. To be fair, the SPACE team [acknowledges](https://queue.acm.org/detail.cfm?id=3454124) there is value in tracking raw development activity (the **_A_** in SPACE) like the kind in McKinsey's Inner and Outer Loops: >>> Developer activity, if measured correctly, can provide valuable but limited insights about developer productivity, engineering systems, and team efficiency. Because of the complex and diverse activities that developers perform, their activity is not easy to measure or quantify. In fact, it is almost impossible to comprehensively measure and quantify all the facets of developer activity across engineering systems and environments. A well-designed engineering system, however, will help in capturing activity metrics along different phases of the software development life cycle and quantify developer activity at scale. But they also clarify that activity tells you nothing on its own without context about the broader systems in place. Activity ≠ Productivity. There are many ways to achieve the goals SPACE and DORA lay out, and how you get there depends on the nature and culture of your organization. For example, a 100% remote workforce of experienced engineers may have different ways of working than a hybrid workforce of various experience levels engaging with contractors around the world. You need to figure out what works best for everyone to encourage diversity in all its forms to deliver the best work, and if you maintain the research-driven SPACE and DORA as your guide and ignore bad faith ruses from mercenaries like McKinsey, you will succeed. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Why Types and Tests are Both Essential in Programming" description: "The arguments for types vs. tests is silly. You need both because they solve two different problems." canonical: https://www.vidyasource.com/blog/why-types-and-tests-are-both-essential-in-programming/ type: article published: 2023-02-26 author: "Neil Chaudhuri" tags: ["Programming", "Software Engineering", "TypeScript", "Python", "JavaScript", "Java", "Agile"] image: https://www.vidyasource.com/img/blog/programming.jpg --- # Why Types and Tests are Both Essential in Programming > The arguments for types vs. tests is silly. You need both because they solve two different problems. - Canonical page: https://www.vidyasource.com/blog/why-types-and-tests-are-both-essential-in-programming/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/why-types-and-tests-are-both-essential-in-programming.json - Published: 2023-02-26 - Author: Neil Chaudhuri - Topics: Programming, Software Engineering, TypeScript, Python, JavaScript, Java, Agile 90% of Tech Twitter is the same arguments over and over again: * Are engineers being laid off simply not good enough? (No. Layoffs are all but random and serve, along with stock buybacks using all the cash on hand tech companies have, to impress The Street and inflate stock price.) * Does preference for remote work mean you're lazy? (No. What matters is if your customers are happy and your business is profitable, and your business should enable you to contribute in the easiest way possible for you and your family.) * Is HTML a programming language? (Who cares? Stop gatekeeping.) * Is Tailwind CSS a productivity boost or the spawn of Satan? (This is stupid. Mind your business. Use what you like and let others do the same.) * Should you have more unit tests or integration tests? (Shoot me.) * Must you love programming to excel? (No. Don't believe me? Then believe [Sarah Drasner](https://leaddev.com/culture-engagement-motivation/why-flow-matters-more-passion), who is smarter than both of us.) Another common debate, and one that is more interesting, is whether you should invest in types or tests. It raises some key questions about what makes each valuable, but unfortunately the debate gets lost in the haze of programming language preferences, developer experiences, and other tangential concerns. Still, the answer matters because it's central to improving our process so we can build as efficiently and with the highest quality as possible. It turns out you need types *and* tests because each solves completely different problems that are both important to you as a developer. ## Types automate API integrity at development time The key components of an API call—whether it's a Java method, a REST endpoint, or a Python function—are the input parameters and the output. There are infinite ways a consumer can interact with your API, but we don't want that. Consider an API call that takes first name, last name, and an amount of money parameters with no types: ~~~typescript (firstName, lastName, amount) => {...} ~~~ Each parameter has literally infinite possible values, so our API forces consumers to be very careful. For example, it's very easy to mix up the `firstName` and `lastName` strings. >>> True story: There is a political fundraising apparatus that has switched my first and last names, so every email starts off "Chaudhuri, we need your help!" when they really mean "Neil, we need your help!" It's like my high school coach is yelling at me to donate. Even worse, they could mix up one of the string values with the number in the amount. In fact, it's very easy to put garbage of any type into each parameter, and relatively speaking almost all of it is bad. Of course you can paper over the problem in a number of ways: * Intuitive parameter names like I did above * Logical order for parameters * Comments or other documentation * Default values or optionality for parameters (if your programming language has those features) These things are good as a general matter, but they don't solve the problem. We can improve the situation considerably by adding types: ~~~typescript (firstName: string, lastName: string, amount: number) => {...} ~~~ Now we have ensured `firstName` and `lastName` can only be strings and `amount` can only be a number. But I know. We can still switch `firstName` and `lastName`. We can put a negative number in for amount. Each parameter *still* has literally infinite possible values, but we have shrunk the universe of valid inputs immeasurably. [Don't let perfect be the enemy of good.](https://en.wikipedia.org/wiki/Perfect_is_the_enemy_of_good) This is progress! We can tighten up our types further to do even better: ~~~typescript (firstName: FirstName, lastName: LastName, amount: Currency) => {...} ~~~ How you do this depends on the features of your programming language. Maybe it has a built-in type already for currency. Or you can define a type with a custom implementation or as simply as a type alias. Regardless of how you get there, look how far we've come. No confusing name with amount. No mixing up names. No improper values for the amount. The tigher our type constraints, the more we make life easier for consumers by narrowing the field of possible values for each parameter. Yes, there are still infinite possible permutations of parameters, but at least it's not "infinite infinite." Types constrain our API to limit the universe of possible inputs, and best of all, they work at development time. If API consumers make a mistake by supplying a value outside the type boundary, they are going to hear about it: from the IDE, type checker, compiler, code generator. We all know the faster you find something wrong the easier it is to fix, and nothing works faster than types to all but guarantee the integrity of our APIs. This is the beauty of automating API integrity rather than wasting time as human compilers and static analyzers who inevitably get it wrong. You also get a form of documentation that helps you understand the API just by looking at it. As for outputs, constraints on return types are also valuable because as we compose functions, the return value from one call becomes the input to another. Programs are essentially data pipelines of composed functions, so types guarantee the integrity of the entire call chain, which is pretty amazing. If you maximize the power of types of the values going in and coming out of your API calls, and you then compose them so they fit together like Legos, you have made enormous strides in maximizing the quality of your code without writing a single test. But that isn't quite enough. ## Tests *verify* your API behavior at runtime We've used types to limit the universe of values our API consumers can apply, but how do we know we have done the right thing with them to satisfy our API contract and return what consumers asked for? Or more simply, does our code work? This is where tests come in. As entire books have been written about testing and you likely already know much about it, I won't spend a lot of time on it here. What matters for this discussion is that while types define APIs in development, tests act at runtime to verify your API behavior with the parameters whose types have already been guaranteed. Types and tests are *complementary*. You need both. ## So why the controversy? While it's been around for as long as static and dynamically typed languages have coexisted, in my experience the Types vs. Tests debate blew up fairly recently in the JavaScript community, where more experienced developers in particular were already comfortable acting as human compilers and static analyzers and viewed the emergence of TypeScript with skepticism. After all, types add more verbosity (to varying degrees given your programming language's talent for type inference), and developers hate extra keystrokes. Add to that more work like `tsconfig.json` for TypeScript or adding hints or the `typing` library in Python and you have a recipe for tension. But at the core of the debate is one key question: "Tests can do everything types can do, so why bother with all the ceremony?" My view: Nah not really. Sure you can fake type safety with tests by enforcing invariants like ensuring `amount`, a `number` in the second example above, is nonnegative. That isn't *actual* type safety though. It's just moving type enforcement from the interface to the implementation, which has disastrous consequences. Like all tests, "type tests" depend on your discipline. You may not write any at all because of deadlines, or you may write bad ones that don't add value. >>> Some might say types demand discipline too. If your types are loose, you don't reap the benefits. That depends on how strongly typed the language is and how effective you are at maximizing the power of its type system. But intentionally widening types is more sabotage than lack of discipline. You've also shifted type guarantees right, from development time to runtime, and delayed feedback as a result. This makes you slower. You know what else makes you slower? You've chosen a manual process over automation. You've sacrificed tooling support and implicit documentation for the chance to waste time doing extra work types do more reliably and cheaply. That's particularly weird since one of the primary objections to types is extra work. In the end, substituting types with tests in your code is like substituting [Chris Hemsworth with his brother Liam](https://www.eonline.com/news/1341657/liam-hemsworth-trolls-perfect-brother-chris-hemsworth-in-marvelous-birthday-tribute) in the cast of your movie. You could do worse but look at how much you've lost in the process. Use types to automate API integrity and tests to verify API behavior *together* to build the most reliable and maintainable code in the shortest amount of time. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Why Coding Interviews are the Worst" description: "Coding interviews are the worst not only for candidates but also for your business. Here's what you should do instead." canonical: https://www.vidyasource.com/blog/why-coding-interviews-are-the-worst/ type: article published: 2022-08-05 author: "Neil Chaudhuri" tags: ["Diversity", "Programming", "Software Engineering", "Open source"] image: https://www.vidyasource.com/img/blog/the-worst.png --- # Why Coding Interviews are the Worst > Coding interviews are the worst not only for candidates but also for your business. Here's what you should do instead. - Canonical page: https://www.vidyasource.com/blog/why-coding-interviews-are-the-worst/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/why-coding-interviews-are-the-worst.json - Published: 2022-08-05 - Author: Neil Chaudhuri - Topics: Diversity, Programming, Software Engineering, Open source Tech interviews are broken. Much like with vegan cheese, we all agree there is a problem, but we all differ on how to fix it. How did we get here? Over the last several decades as tech became a dominant industry, the new personalities that became very rich very fast also became very [Gavin Belson from Silicon Valley](https://silicon-valley.fandom.com/wiki/Gavin_Belson)—supremely convinced of their own outside-the-box cleverness not only at producing revolutionary technology but also at revolutionizing the way it's built. The media decided this was the beginning of a new era where the novel approaches tech companies took to everything should be a model for everyone else. Remember when open floor plans with foosball tables and Keurigs were the key to unleashing workplace productivity restrained too long by dehumanizing cubicles? After enough writeups in even mainstream outlets like [CBS News](https://www.cbsnews.com/news/ditch-the-cubicles-for-better-collaboration/), everyone was doing it. But we ended up just using headphones to fashion our own mental cubicles so we could work in some kind of awkward peace—[particularly awkward for women](https://www.inc.com/betsy-mikel/new-study-open-offices-are-terrible-for-women.html), people of color, introverts, and many others. A similar thing happened with tech interviews. After all, boring accounting firms ask about your resume, but not us! The media were awestruck at profound interview questions like "*Why are manhole covers round?*" or "*How would you calculate the number of cars passing through a busy bridge?*" or the famous Elon Musk riddle "*You’re standing on the surface of the Earth. You walk one mile south, one mile west, and one mile north. You end up exactly where you started. Where are you?*" This is all about getting deeper than any traditional interview could, you see, to reveal how candidates think! Yeah, OK. ## Coding interviews are the worst. How do we know? While the more absurd questions have lost some luster, one relic of our unconventional interview style that endures is the coding interview. You know it. A panel of dudes all vaguely reminiscent of [Howard](https://bigbangtheory.fandom.com/wiki/Howard_Wolowitz) from *The Big Bang Theory* asks candidates to go to the whiteboard or a laptop broadcasting to a screen and write code for something they've probably never seen (by design). It could be implementing a doubly-linked list or merge sort or a function that determines if a string is a palindrome. It could be performing a breadth-first search or converting a loop into a recursive function. And on and on. This is a problem—not only for job candidates but also for the companies considering them. It's an especially acute problem now with the shortage of software engineers and the popularity of remote work particularly in the aftermath of a global pandemic. Then there is all the uncertainty. Uncertainty tech companies feel in an era of inflation and [evolving economic fundamentals](https://www.thestar.com/business/2022/07/26/i-got-this-wrong-shopify-ceo-announces-plan-to-layoff-10-per-cent-of-staff.html) and uncertainty candidates feel in an era of layoffs and evolving personal responsibilities and professional goals. ### Coding interviews solve for the Computer Science Department valedictorian or the most short term knowledge. Not the best long term culture fit The biggest technical problem with coding interviews is the disconnect between what they measure and what actually adds value to the business. For 99% of you, hiring someone who understands Kruskal’s algorithm for finding a minimum spanning tree isn't relevant to making your UI accessible, scaling in the cloud, improving your engineering practices, or driving revenue. Even if you do better than silly college computer science questions and focus on the tech you use today, you focus on the wrong thing. It isn't what candidates know; it's how much they can learn. If you're a React shop, it's impressive if a candidate understands how `useEffect` is about [syncing UI with state](https://www.vidyasource.com/blog/dark-mode-nextjs-tailwindcss-react-hooks/) rather than a new spin on `componentDidMount`, but what if you decide to move to [Solid](https://www.solidjs.com/)? Or what if an important new initiative demands a different set of skills entirely and you don't have the resources to hire a subject-matter expert right away? Instead, you want to hire engineers who fit with your culture and make their teams better with hard and soft skills. They do this in many ways. They learn quickly because they think in abstractions so they can translate prior knowledge into new skills that serve their professional growth and your bottom line. They aren't afraid to experiment and fail in order to learn new things. They listen. They are eager to help. They are patient and kind. They manage their time well. They can communicate with customers. They care about good code and write good documentation. Their teammates trust them to support them and to assume leadership when the moment calls for it. Your business needs [Vision](https://marvelcinematicuniverse.fandom.com/wiki/Vision). Coding interviews at best get you [Ultron](https://marvelcinematicuniverse.fandom.com/wiki/Ultron). ### Coding interviews further exclude systematically excluded communities I am a passionate advocate for diversity in tech. Prejudice against anyone on the basis of race, gender, ethnicity, nationality, religion, orientation, ability, or even academic background not only limits our individual growth but also limits the creative energy necessary to develop the best software. Sadly, there are countless institutional barriers to diversity in tech, and coding interviews are one of them. Don't believe me? Check out this [study by North Carolina State University and Microsoft](https://news.ncsu.edu/2020/07/tech-job-interviews-anxiety/): > But the format may also serve as a barrier to entire classes of candidates. For example, in our study, all of the women who took the public interview failed, while all of the women who took the private interview passed. Our study was limited, and a larger sample size would be needed to draw firm conclusions, but the idea that the very design of the interview process may effectively exclude an entire class of job candidates is troubling. Much as society writ large is convinced [DNA evidence](https://www.youtube.com/watch?v=ScmJvmzDcG0) and [AI](https://www.theguardian.com/technology/2016/mar/24/tay-microsofts-ai-chatbot-gets-a-crash-course-in-racism-from-twitter) are "objective," tech is at best convinced coding interviews are also free from bias or at worst happy to capitalize on their bias. Bias in coding interviews could be favoritism, intentional or otherwise, reflected in choice of coding problem or leniency in evaluation. They are also biased against realistic forms of working. In real life, we don't have people staring at us to scrutinize our work and mannerisms as we type. We have access to Google, Stack Overflow, GitHub Copilot, and other resources. The discomfort produced by contrived awkwardness in coding interviews hurts otherwise talented candidates—particularly those in communities already systematically excluded in tech like the women in the study. But that's one study you say. OK, but consider anecdotal evidence from elite engineers. [Jenn Creighton](https://twitter.com/gurlcode), senior engineer at Netflix on the NodeJS team and host of the [single-threaded podcast](https://twitter.com/single_threaded) exposed the problem on her premier episode with her guest [Erin Fox](https://twitter.com/erinfoox). Senior engineer, Apple whistleblower, and activist [Cher Scarlett](https://twitter.com/cherthedev) described her own experience a few years ago: > not work, and wasn't what they were looking for. These same two pieces of code are now written about by two other individuals as best practice solutions to these problems. Do you think this coding interview was about code? No, it was an ego battle I'd never be allowed to win.— Cher Scarlett (@cherthedev) [May 22, 2018](https://twitter.com/cherthedev/status/998914080177115136?ref_src=twsrc%5Etfw) Of course if things can get so bad for cis white women in tech that coding interviews become a vanity exercise at their expense, just imagine how much worse it must be for women of color, trans people, anuerotypical people, even people who learned tech at boot camps instead of MIT or have degrees in psychology. And so many others. Let's use the interview process to welcome as many people as possible into tech because the challenges that lie ahead are too great to leave anyone behind. ### Coding interviews are the basis for an entire cottage industry dedicated to help candidates "ace" them Precisely because coding interviews are so divorced from the reality of our lives as software engineers, job candidates need time to prepare for the experience. It takes time to build composure as people stare at you at the front of the room or judge every keystroke. It takes time to remember, or learn for the first time, fundamental concepts of computer science or low-level implementation details typically abstracted from you by frameworks and open-source libraries. It takes time to memorize it all because you may not be allowed access to the resources you normally have at your disposal. Everyone knows it, so a cottage industry has emerged to get candidates through it. There are literally hundreds of books and online courses promising to help them "ace" the tech interview and get that prestigious, high-paying job so you can get that Cybertruck because you find its ugliness endearing. This is costly. At best, it takes time away from personal life. At worst, it costs serious dollars. Either way, this is another way coding interviews serve as a gatekeeper excluding communities from tech—software engineers who can't afford the costs of interview prep. ## What's the alternative? Coding interviews are bad for the business and bad for the candidate. Instead, you should conduct interviews that are inclusive, find the best long term fit for your business, and convey the joy of solving problems there. Make the interview a conversation. Share your business goals and details about your mission, the role you're hiring for, your expectations technically and culturally, and your values as an organization. Then consider questions like these. ### "What are some examples of problems you solved at work? Take me through how you solved them." This question helps you understand the experiences candidates have had on the job. It helps you evaluate how these experiences apply to your business and, more importantly, how your candidates approach challenges. Feel free to explore the topic in depth and follow threads of conversation as candidates raise points you find particularly interesting. You don't have to be [Oprah](https://www.youtube.com/watch?v=w-gkAM0XZMU), but there is an art to navigating a conversation to elicit more information. You will get a good sense of candidates' technical knowledge, their creativity, and their ability to gain new knowledge. After all, game recognizes game. ### "Can you describe a situation at work where you were particularly proud of what you accomplished?" This question helps lighten the mood and brings positivity into an otherwise tense process. It gives candidates a chance to brag a little but in a way that lends value to you. You can glean if your own business offers the kinds of opportunities for joy the candidates want to experience, and you can again get some detail into their experience, expertise, and potential for growth. ### "It's common to have differences of opinion on a team on how to solve problems. How have you worked through those?" This question addresses the reality of working on a team and the likelihood of a culture fit. Anyone who spends about three minutes on Tech Twitter knows we like to argue—often about the most trivial, self-serving things. Candidates have certainly had their share of disagreements at work, and it's important to understand how constructively they handle them and if their solutions fit with your company culture. You will also get a sense of what candidates think is important—tech or otherwise. The question is if your value system is compatible with theirs. ### "If you had full control of the architecture and engineering of your current code base, what would you change about the code or the process?" This question again helps you learn about a candidate's expertise and room for growth but from a different angle. We all have regrets about technical debt, yearnings for different approaches, and other aspects of the product and process that stick in our craw. Give candidates a chance to vent. In the process, you will learn about what candidates value, their knowledge of the engineering and product delivery landscapes, the direction they want to go, and their tact with constructive criticism. Does their vision comport with yours for your business? ### "What kinds of support do you need to do your best work?" This question addresses *your* responsibility to your people to achieve the goals of the business and the support that candidates can expect if they join. You might have an issue with this question because it implicitly undermines conventional wisdom that the best engineers do their best work of their own accord if they have "passion" for tech implied by a robust GitHub presence, lots of side projects, eagerness to work weekends, etc. Nope. For reasons why the responsibility is yours to provide "flow state" rather than candidates' responsibility to have passion, read [this excellent post by Sarah Drasner](https://leaddev.com/culture-engagement-motivation/why-flow-matters-more-passion), Director of Engineering, Core Developer Web at Google. The best engineering managers understand how to create flow state. Even if candidates are unfamiliar with management theory, this question informs them that you have their best interests at heart and recognize your own responsibility in making them successful. You can also evaluate whether their answers are aligned with your business values. ### "Do you have any questions for us?" Hopefully you are asking this question anyway regardless of whether coding interviews are part of your hiring process. Give candidates a chance to ask you about the things that matter to them as they ponder making the considerable, potentially life-changing commitment to your business. They should direct this part of the conversation. Expect questions about titles, commitment to and evidence of diversity, opportunities for advancement, remote work and tooling, and other aspects of business process and company culture. Be careful though not to interpret the absence of questions as apathy or really anything negative. This is their time, and if they don't need it, you shouldn't let that influence your decision one way or another. > As an aside, prefer diverse panels of interviewers. Also pay attention to the demeanor of candidates. Any toxic behavior like arrogance, abruptness, or something particularly awful like refusing to acknowledge female interviewers should make your decision easy. ## But OK, if you must do coding interviews... Let's face it. Many of you will never accept that coding interviews are the worst. The idea is simply too entrenched in tech. You're convinced instant recall of computer science fundamentals is critical for bringing value to your business. Besides, tech is full of imposters! You must expose them with cleverly chosen coding exercises they have never seen lest they compromise your business with their duplicity. There is no evidence any of that is true, but it sure [feels true in your gut](https://www.c-span.org/video/?c4293026/user-clip-stephen-colbert-truthiness). The funny thing is there are actually a lot of job candidates who also think it's true. They see coding interviews as a perfectly reasonable expectation and happily, if often nervously, prepare accordingly. Fine. If you must, at least adapt your coding interviews like this: * Offer candidates the option either to live code or to do a short, paid(!) project to do at home. * Have them solve a real but non-critical and non-sensitive problem you have at work. * If it's a live coding exercise, collaborate with candidates rather than sit back and watch, and give them access to resources like AI. We have a lot of important work ahead of us as an industry. I hope you consider the lessons I've learned to help your business grow to meet the challenges ahead. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "SWaM Certification: Vidya Is Certified in Virginia" description: "Vidya earned SWaM certification from the Virginia Department of Small Business and Supplier Diversity as a Small, Micro, Minority Owned, and 8(a) business." canonical: https://www.vidyasource.com/blog/vidya-is-swam-certified-by-virginia-department-small-business-supplier-diversity/ type: article published: 2022-07-07 author: "Neil Chaudhuri" tags: ["Partners", "Diversity", "Programming", "Software Engineering", "Agile", "Lean", "Government"] image: https://www.vidyasource.com/img/blog/swam-min.jpeg --- # SWaM Certification: Vidya Is Certified in Virginia > Vidya earned SWaM certification from the Virginia Department of Small Business and Supplier Diversity as a Small, Micro, Minority Owned, and 8(a) business. - Canonical page: https://www.vidyasource.com/blog/vidya-is-swam-certified-by-virginia-department-small-business-supplier-diversity/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/vidya-is-swam-certified-by-virginia-department-small-business-supplier-diversity.json - Published: 2022-07-07 - Author: Neil Chaudhuri - Topics: Partners, Diversity, Programming, Software Engineering, Agile, Lean, Government I am excited to announce that hot off the heels of [8(a) certification by the US Small Business Administration](https://www.vidyasource.com/blog/vidya-is-8a-certified-by-us-small-business-administration/), the [Commonwealth of Virginia Department of Small Business and Supplier Diversity (SBSD)](https://www.sbsd.virginia.gov/certification-division/swam/) has now also certified Vidya as part of its Small, Women-owned, and Minority-owned Business (SWaM) program. At Vidya, we have been very privileged to deliver high-quality software to a wide array of government and commercial clients using exciting technology while promoting lean principles, agile software development, and diversity in software engineering. The best part about it? It's been fun! Doing great work and contributing to the community—whether writing on [our blog](https://www.vidyasource.com/blog/), answering questions on [Stack Overflow](http://stackoverflow.com/users/1347281/vidya), contributing to open-source on [GitHub](https://github.com/VidyaSource), speaking at [conferences](https://www.vidyasource.com/blog/speaking-at-code-writers-workshop-2017/), and even posting video tutorials on [YouTube](https://www.youtube.com/channel/UC24LVc8Bb65SF6LW-SLog9A)—have been the most rewarding professional experience of my life. Vidya is based in Virginia just outside Washington, DC, and SWaM certification by SBSD shows Virginia that Vidya has demonstrated the kind of growth potential that should make state government agencies who need technology services take notice. As the SBSD [describes](https://www.sbsd.virginia.gov/certification-division/swam/): > The purpose is to enhance procurement opportunities for SWaM businesses participating in state-funded projects. We look forward to building new relationships and to delivering our services to government agencies in the Commonwealth of Virginia. We are also excited about expanding our current offerings by making our courses available online and building new software products. We want to thank the Virginia Department of Small Business and Supplier Diversity for recognizing Vidya's past success and future potential, and we cannot wait to do business with you. [Let's talk](https://www.vidyasource.com/contact/). ![Virginia Department of Small Business and Supplier Diversity Small, Micro, Minority Owned, and 8(a) Certified](https://www.vidyasource.com/img/certifications/swam.jpeg) --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Vidya is 8(a) Certified!" description: "We are proud to be certified 8(a) by the US Small Business Administration." canonical: https://www.vidyasource.com/blog/vidya-is-8a-certified-by-us-small-business-administration/ type: article published: 2022-05-01 author: "Neil Chaudhuri" tags: ["Partners", "Diversity", "Programming", "Software Engineering", "Agile", "Lean", "Government"] image: https://www.vidyasource.com/img/blog/sba.png --- # Vidya is 8(a) Certified! > We are proud to be certified 8(a) by the US Small Business Administration. - Canonical page: https://www.vidyasource.com/blog/vidya-is-8a-certified-by-us-small-business-administration/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/vidya-is-8a-certified-by-us-small-business-administration.json - Published: 2022-05-01 - Author: Neil Chaudhuri - Topics: Partners, Diversity, Programming, Software Engineering, Agile, Lean, Government I don't have to tell you it's been a difficult few years (for all of us!), so I am absolutely thrilled to announce [US Small Business Administration (SBA)](https://www.sba.gov/) has certified Vidya as an [8(a) business](https://www.sba.gov/federal-contracting/contracting-assistance-programs/8a-business-development-program). At Vidya, we have been very privileged to deliver high-quality software to a wide array of government and commercial clients using exciting technology while promoting lean principles, agile software development, and diversity in software engineering. The best part about it? It's been fun! Doing great work and contributing to the community—whether writing on [our blog](https://www.vidyasource.com/blog/), answering questions on [Stack Overflow](http://stackoverflow.com/users/1347281/vidya), contributing to open-source on [GitHub](https://github.com/VidyaSource), speaking at [conferences](https://www.vidyasource.com/blog/speaking-at-code-writers-workshop-2017/), and even posting video tutorials on [YouTube](https://www.youtube.com/channel/UC24LVc8Bb65SF6LW-SLog9A)—have been the most rewarding professional experience of my life. 8(a) certification by SBA shows the nation that Vidya has demonstrated the kind of growth potential that should make government agencies who need technology services take notice. As they [describe](https://www.sba.gov/about-sba): > SBA is the only cabinet-level federal agency fully dedicated to small business and provides counseling, capital, and contracting expertise as the nation’s only go-to resource and voice for small businesses. We look forward to building new relationships and to delivering our services to even more government agencies. We are also excited about expanding our current offerings by making our courses available online and building new software products. We want to thank SBA for recognizing Vidya's past success and future potential, and we cannot wait to do business with you. [Let's talk](https://www.vidyasource.com/contact/). ![US Small Business Administration 8(a) Certified](https://www.vidyasource.com/img/certifications/8a.png) --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Which Programming Languages Should I Learn First?" description: "When you're just starting out in tech, don't focus on programming languages but on something else entirely." canonical: https://www.vidyasource.com/blog/which-programming-languages-should-i-learn-first/ type: article published: 2022-02-03 author: "Neil Chaudhuri" tags: ["Programming", "Big Data", "Open Source", "Web development"] image: https://www.vidyasource.com/img/blog/programming.jpg --- # Which Programming Languages Should I Learn First? > When you're just starting out in tech, don't focus on programming languages but on something else entirely. - Canonical page: https://www.vidyasource.com/blog/which-programming-languages-should-i-learn-first/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/which-programming-languages-should-i-learn-first.json - Published: 2022-02-03 - Author: Neil Chaudhuri - Topics: Programming, Big Data, Open Source, Web development People who want to get into programming often ask me which languages to learn first, but as Dee tells Naomi on the series premier of her [eponymous TV show](https://www.cwtv.com/shows/naomi/dont-believe-everything-you-think/?play=109fca5a-4aec-411d-90e6-19aa22eaedc6), "I can’t give you answers when you’re not asking the right questions." I think it's more helpful to identify which problems you want to solve first. Do you want to build a website to sell your mom's amazing ghee? Look at Shopify, Squarespace, or Wix for a low- or no-code approach or Next.js or Remix Run for a bespoke site. Do you want to build a data model that can help American football coaches decide whether they should go for it on 4th down? Look at Anaconda or Tensor Flow Lite. Don't let me stop you if you're jonesing to learn a particular programming language. But if you take my advice and come at the learning process from the point of view of providing a real business need (Your own business need counts!), figure out which tools, libraries, and frameworks make the most sense as you try to deliver for yourself and learn something along the way. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "What If...Marvel Built a Minimum Viable Product?" description: "Product development should begin with an MVP, and Marvel's What if?... is exactly that." canonical: https://www.vidyasource.com/blog/what-if-marvel-built-a-minimum-viable-product/ type: article published: 2021-11-09 author: "Neil Chaudhuri" tags: ["Agile", "Lean", "Project Management", "Software Engineering"] image: https://www.vidyasource.com/img/blog/what-if.jpeg --- # What If...Marvel Built a Minimum Viable Product? > Product development should begin with an MVP, and Marvel's What if?... is exactly that. - Canonical page: https://www.vidyasource.com/blog/what-if-marvel-built-a-minimum-viable-product/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/what-if-marvel-built-a-minimum-viable-product.json - Published: 2021-11-09 - Author: Neil Chaudhuri - Topics: Agile, Lean, Project Management, Software Engineering In 2011, Eric Ries published *The Lean Startup*, and it revolutionized software development. Lean principles had already [made it into agile software development](https://www.goodreads.com/book/show/194338.Lean_Software_Development), but *The Lean Startup* fills in a lot of gaps to turn the abstract [principles behind the Agile Manifesto](https://agilemanifesto.org/principles.html) into something real. For example, even if you nailed agile and built the thing right, how do you know you built the right thing? In other words, it's great to execute on your project and build exactly what you intend on time and on budget, but when you're done, how do you know people will actually buy it? Although the Agile Manifesto never answers this question because it assumes you are on the right track all along, Ries gives us an answer. ## The Minimum Viable Product Probably the most profound innovation in *The Lean Startup* is the Minimum Viable Product (MVP). (You can [watch Ries himself talk about it](https://www.youtube.com/watch?v=E4ex0fejo8w), but I have to warn you that it looks like it was filmed by the director of [The Blair Witch Project](https://www.thenewsminute.com/article/how-blair-witch-project-manages-scare-without-showing-anything-supernatural-135492).) The idea is really quite simple. Whether you are conscious of it or not, the product you have in mind is based on your assumptions about what your customers want and why, so the purpose of the MVP is to validate those assumptions with hard data around *a lightweight representation of your product* before you commit your full resources to building it out in full. You give your early adopters access to the MVP and consider their feedback, which Ries calls "validated learning." Validated learning is crucial. For Ries, it's quantifiable feedback that is "the unit of progress for lean startups." Based on what you learned, you either "[pivot](https://www.youtube.com/watch?v=n67RYI_0sc0)" to something else more aligned to what your customers want or "persevere" because you had it right all along. From this point on, you have quantifiable, proven demand for your product, and you can be confident that you will be investing in building the right thing. Like a lot of terms in tech that blow up, consultants often use MVP to mean something different from what Ries intended--most commonly to mean the first version of your product and/or a really scaled down version of it. In either case you've implicitly decided without any validated learning what you want to build. The fact it's in its earliest stages is irrelevant. So what does all of this have to do with Marvel? ## The MCU Juggernaut Unless you have been living in the [Quantum Realm](https://marvelcinematicuniverse.fandom.com/wiki/Quantum_Realm) for the last decade, you know the [Marvel Cinematic Universe (MCU)](https://www.marvel.com/movies). It's a monumental achievement in cinema that has taken decades of Marvel comics and reformulated them into a multibillion dollar juggernaut movie franchise that has raised once second-tier heroes like Iron-Man, Thor, and Black Widow to the stature of eternal favorites like Spider-Man, Captain America, and Hulk and put [Mjolnirs](https://marvel.fandom.com/wiki/Mjolnir) and [Infinity Gauntlets](https://marvelcinematicuniverse.fandom.com/wiki/Infinity_Gauntlet) into the homes of ardent fans worldwide. It may be hard to imagine now, but there was no guarantee of success at the beginning as Marvel was struggling as a business having sold off their most bankable assets. This forced Kevin Feige, the architect of the MCU, to get creative to and [derive new ways to bring Marvel heroes to life](https://www.vox.com/2016/5/9/11595344/marvel-cinematic-universe-captain-america-avengers). They succeeded spectacularly, and now they have to do it all over again. Firmly entrenched in global pop culture, the MCU now faces pressure to build on its foundation with new heroes and villains, many of whom like [Moon Knight](https://marvel.fandom.com/wiki/Marc_Spector_(Earth-616)) and [Titania](https://www.marvel.com/characters/titania-mary-macpherran) lack the star power of Captain America and Spider-Man and are all but unknown to mainstream audiences. And considering that the MCU has decided to go all in on the [Multiverse](https://marvelcinematicuniverse.fandom.com/wiki/Multiverse), there are literally infinite possibilities for stories. How can Marvel decide what direction to take with the MCU? ## *What If?* is Marvel's MVP There is no question Kevin Feige has sketched out the broad contours of the next iteration of the MCU. Still, as sterling as his record is, he's working from assumptions about what will make for great stories and what will thrill audiences going forward. If he followed *The Lean Startup* model, Feige would build a lightweight MVP to test his hypotheses with passionate fans and measure their response for some validated learning. It turns out that is exactly what Marvel is doing with *What If?...* *What If?...* is an [animated show on Disney+](https://www.marvel.com/tv-shows/animation/what-if/1) that imagines the MCU heroes we know experiencing very different lives throughout the multiverse. For example, we see Peggy Carter receive the super soldier serum rather than Steve Rogers to become [Captain Carter](https://www.marvel.com/articles/tv-shows/what-if-episode-1-multiverse-report-captain-carter), and we watch somberly as T'Challa, voiced just as he is portrayed on screen by the legendary Chadwick Boseman in his final performance, [becomes Star-Lord](https://www.marvel.com/articles/tv-shows/what-if-new-images-episode-2) rather than Peter Quill. It's heroes and villains we know so well from the MCU facing new challenges alongside other heroes and villains we've never seen them interact with before. The novel combinations are fun and exciting, but they're also an experiment. How will fans, primarily hardcore MCU fans (early adopters if you will) react? You can easily track this with viewership metrics on Disney+, mentions on Twitter, and the impressions of media influencers for validated learning. When a story doesn't click, Marvel can dismiss it as a fun idea for the fans and pivot to something else. On the other hand, when a story does click, Marvel can persevere and explore it further on a live-action series on Disney+ or even on the big screen within established MCU canon. Best of all, Marvel can test their hypotheses, engage in validated learning, and guide the future of the MCU accordingly at a low cost. *What If?...* is high-quality animation, and most of the elite talent that portrays the characters on the big screen voices them on the show. Still, the cost of running these experiments in animation without any real commitment since the stories all happen outside the "prime" MCU universe is orders of magnitude less than it would be going all in with any of the *What If?...* plots in live action. ## The Key Lesson One way or another, we are all in the business of product development. Too often we assume we have accurately gauged customer sentiment, and [we exude hubris that success is inevitable only to fail miserably](https://www.youtube.com/watch?v=t-_PfdQ0DYo). Don't be afraid to test your assumptions before you go all in. Build a Minimum Viable Product--something more lightweight and less expensive that will nonetheless provide your early adopters something meaningful to evaluate. The validated learning you get from their feedback will either help you pivot to something more aligned with what people want or empower you to persevere with your original idea with legitimate confidence it will sell. Once you start executing, then you can argue on Twitter about whether you need daily meetings and whether people should be allowed to sit down in them. Using an MVP to build confidence in your product direction is exactly what the Marvel Cinematic Universe has done with *What If?...*, but you don't have to be Marvel to fly [higher, further, faster, baby](https://www.youtube.com/watch?v=eKAvj9EjBmM). --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Lessons Learned from Building a React Component Library with TypeScript" description: "Lessons learned, and not only about tech, from building a React component library for government." canonical: https://www.vidyasource.com/blog/lessons-learned-react-component-library-typescript/ type: article published: 2021-10-11 author: "Neil Chaudhuri" tags: ["React", "TypeScript", "Chakra UI", "Accessibility", "Vite", "Jest", "React Testing Library", "Storybook", "Open source", "Government"] image: https://www.vidyasource.com/img/blog/react-ts.png --- # Lessons Learned from Building a React Component Library with TypeScript > Lessons learned, and not only about tech, from building a React component library for government. - Canonical page: https://www.vidyasource.com/blog/lessons-learned-react-component-library-typescript/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/lessons-learned-react-component-library-typescript.json - Published: 2021-10-11 - Author: Neil Chaudhuri - Topics: React, TypeScript, Chakra UI, Accessibility, Vite, Jest, React Testing Library, Storybook, Open source, Government Component libraries are all the rage. Shopify, Salesforce, IBM, and even the [United States government](https://designsystem.digital.gov/components/overview/) have joined countless other organizations and businesses in building component libraries. They're the subject of blog posts, podcasts, and YouTube tutorials. All that's left is a [Ken Burns documentary](https://kenburns.com/the-films/) on the subject. In fact, I am a software architect and senior engineer, and I currently lead the development of a React component library that will be the basis for the UIs for a prominent US government agency. I want to share with you my lessons learned in project management, communications, accessibility, engineering, and testing to build something that will impact the lives of millions. And the ups and downs of it all. So what's the big deal with component libraries? ## The Design System It doesn't start with a component library; it starts with a design system. The Nielsen Norman Group defines design systems [this way](https://www.nngroup.com/articles/design-systems-101/): > A design system is a complete set of standards intended to manage design at scale using reusable components and patterns. A design system enumerates the standards and practices that comprise the premier UX for consumers of your brand. It expresses the nomenclature every team should use in communications to break down silos and avoid the impulse from [Conway's Law](https://www.melconway.com/Home/Conways_Law.html). There are basic rules about colors, typography, spacing, and so on. All of these core principles become the basis for larger components--explicit ones like buttons and date pickers and subtler ones like grid systems. Our UX team develops and maintains our design system. Like software, it evolves; it's versioned; and it's collaborative. There are conversations among the UX designers and with me and other architects and engineers on the program about what makes sense and what is feasible. Are nested dropdowns necessary? Do we have time to create our own perfect `Datepicker`? Or do we try to customize something open source? How do we feel about disabled buttons, and if we think they make sense, how can we overcome common pitfalls like poor [contrast ratios](https://developer.mozilla.org/en-US/docs/Web/Accessibility/Understanding_WCAG/Perceivable/Color_contrast)? Stuff like that. We use the language of [Atomic Design](https://bradfrost.com/blog/post/atomic-web-design/), which deconstructs web interfaces into entities ranging from "atoms" to "pages," as a common nomenclature to describe the goals of the design system. The challenge, and probably the hardest part of building a component library for us, is the tooling. Partly because of the preferences of the UX team and partly because of constraints on our development environment due to the sensitive nature of our work, we have not been able to streamline automation for versioning UX wireframes or translating them into artifacts engineers can use to build. As a result, we work with wireframes that are cumbersome to understand. In order to even view them, we either need to install the tool on our machines, which costs more licenses and imposes a burden on developer experience (DX), or we need to wade through literally hundreds of static asset files with a custom browser plugin. Neither is an optimal experience. Beyond that, it's a manual process to track consistency between the design system and the component library as both evolve. I never said it was pretty, but it isn't all bad either. ## The Value of a Component Library The design system is a set of core principles independent of implementation details. You can choose to implement these principles and make them real for UI engineers with whatever technology you choose. For us, that's React. Our React components generate a lot of value for the program. ### Consistency Our component library enforces our design system across our development teams. Using the components all but guarantees a UI will be consistent with our brand and provide our users the best, most intuitive experience. Developers can feel confident they are using components vetted with the UX team, which frees them up to work on the specific use cases of their services rather than cross-cutting concerns like consistency with the design system. The library also maximizes the likelihood that our UIs pass visual testing by our UX team. This is important as violations slow down our delivery cadence and ability to get feedback. ### Accessibility Related to consistency is accessibility, which is a first-class priority for our component library. Accessibility, commonly known as [#a11y](https://www.a11yproject.com/), is more than just empowering the visually impaired. It also means empowering people who experience difficulty with hearing, motion, dexterity, or anything else. It means empowering *everyone*. The program is required by contract and [by law](https://www.access-board.gov/law/ra.html#section-508-federal-electronic-and-information-technology) to produce UIs that are accessible--specifically [508 compliance](https://www.section508.gov/tools/playbooks/technology-accessibility-playbook-intro/). That said, accessibility is far more than a professional obligation; it is my personal priority. It is very important to me that everything I build is intuitive for every user. I will elaborate on this shortly, but our component library is built for accessibility. Development teams can trust the accessibility of the individual components, and as I said before, focus on their own use cases. Of course you are probably thinking in terms of accessible dropdowns and autocompletes and datepickers, which we have, but we also provide helper [Semantic HTML](https://developer.mozilla.org/en-US/docs/Glossary/Semantics#semantics_in_html) components. For example, the library features `Section`, which represents the `section` [HTML element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/section) as you would imagine, and `SectionGrid`, which is a `section` element endowed with our design system grid. Of course, the component library can only take developers part of the way to full accessibility, but it's nice not to have to start from 0. ### Reusability We have worked very hard to provide intuitive APIs for our components, but the task is trickier than you might think. The APIs need to impose enough opinion so that consumers don't violate the design system but allow enough freedom for the components to support a wide range of use cases. For our `Button` component, that is easy enough. For layout components like `Card` and `Page`, it's tougher. The reusability that results has made individual teams and the entire program so much more productive. We also go out of our way to endow our components with as little functionality as possible. Component APIs offer props that enable library consumers on the development teams to supply behavior. For an obvious example, developers supply `onClick` behavior to the `Button` component. We have more complex components that need to maintain their own state, but we try to minimize that where possible. This provides a clean separation of concerns, which makes testing our components much easier, and anyone who has been in the game long enough knows that strong testability makes for strong reusability. ### Encapsulation There will be more about this shortly, but we do not build our components from scratch. Rather, we customize existing open source components and map our APIs to theirs. This abstracts the implementation details of the component from our development teams. For example, we use [react-datepicker](https://github.com/Hacker0x01/react-datepicker) as the basis for our own `DatePicker`, but if we decide to swap it out for a different one, our consumers will be none the wiser. ## Component Stack As I mentioned, we build our component library with React, which is what we recommended but is also, for our risk-averse government customer, the safe choice given its backing by Facebook, [its market penetration](https://insights.stackoverflow.com/survey/2021#section-most-popular-technologies-web-frameworks), and [its popularity](https://insights.stackoverflow.com/survey/2021#most-loved-dreaded-and-wanted-webframe-want). But React is the easy part. Let's look at other parts of the component stack. ### TypeScript When we started building the component library, I considered TypeScript essential for two reasons. By enforcing type safety during development and at build time, we catch bugs much faster, which from a project management standpoint is much cheaper. More importantly, building our APIs in TypeScript is a huge help to library consumers on application development teams by facilitating code completion in their IDEs and type checking in *their* builds. Let me also mention that some of our TypeScript APIs require [ARIA](https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA) values to promote accessibility if we can't derive them ourselves from other props. ### Chakra UI I mentioned earlier that our components are built on open source components, and most of them are built on [Chakra UI](https://chakra-ui.com/). There are many other open source component libraries out there, but Chakra UI is my favorite by far. The primary reasons are its first-class commitment to accessibility and the intuitive APIs of its components built with TypeScript. As you can probably infer, Chakra UI is an inspiration to me when building our own component library on top of it. Chakra UI also offers a powerful [theme customization API](https://chakra-ui.com/docs/theming/customize-theme) we leverage heavily to apply the principles of our design system to Chakra components via dedicated theme files that separate the styling from functionality. This separation of concerns makes it easier to reason about our code and makes the files themselves a lot lighter. Chakra UI also features with some helpful hooks like [useDisclosure](https://chakra-ui.com/docs/hooks/use-disclosure) that come in handy. If you use Chakra UI for your own component library, you will probably need some alias imports to deal with name collisions. For example, we call our button components, to no one's surprise, `Button`, but so does Chakra UI. So we do this: ~~~js import { Button as ChakraButton } from "@chakra-ui/react" ~~~ ## Engineering Of course the fun part is building a React component library. This post is long enough, so I can't get into every detail. But I do want to address some of the key aspects you might want to consider when you build your own. ### Workflow When we first began building the component library, we needed to move quickly because development teams were waiting on us to start building their UIs. Our management tasked me and several developers to get something done in a few sprints at nearly a full time commitment. We got the initial design system specification from the UX team and got to work. After those first few sprints, we had built enough components to allow teams to get going. The problem is that all of us resumed our normal duties with no time allocation for the library. This meant that whenever the UX team designed new components or developers found bugs in existing components, there was a bottleneck because no one was dedicated to upgrading the library. I and others got to it when we could, but the absence of a dedicated team was a problem. Another problem is the initial lack of communication within the UX team itself and among the UX team, developers, and me. In their creative zeal, far too often they provided wireframes to some developers inconsistent with wireframes provided to others, or they provided wireframes featuring components that weren't in the library. Development teams assumed they *were* in the library and estimated accordingly. As you might expect, they were unhappy when they discovered the components didn't exist, which impacted their ability to deliver on schedule. They let me know it, and frankly they had every right to be unhappy. I knew we had to improve our process. To that end, we made some changes. We established a Microsoft Teams channel to encourage communication by eliminating the ceremony of meetings and even E-mails. We also decided that development teams will build new components initially, and if other teams will benefit, the library will absorb them, with tweaks as needed to APIs or implementations, to support broader applicability across the program. Then the team that built the component first will replace their implementation with the library's when ready. While this means teams have to devote more time to developing components, it's transparent, and there is no bottleneck. This is an evolving workflow. There is always room for improvement. ### Component structure Our components in TypeScript take three forms. The simplest components look like this: ~~~js export const TimePicker = (p: TimePickerProps) => { ... } ~~~ Our `TimePicker` component has no children, so it's as straightforward as it gets. It's just a function! If the component has children, it still isn't too bad: ~~~js export const Card: React.FC = p => { ... } ~~~ React's `FC` type (for `FunctionComponent`) includes a `children` prop implicitly. We could also declare it just as we do `TimePicker` but explicitly add a `children` prop of type `ReactNode` to `CardProps`. I prefer `FC` because it very clearly signifies the presence of `children` to library consumers and because the type parameter lets me enjoy some type inference. Notice how I don't have to specify the type of `p` because it's implicit from the type parameter `CardProps`. Still, not too bad, right? The last kind of component is a little complicated--form components. Our developers use [React Hook Form](https://react-hook-form.com/), and like every other form library I've used, it uses `ref`s to maintain form state. This means our components need to provide a way to accept a `ref` and delegate it to their children. Most React engineers don't know this because they don't have to, but React provides a function for exactly this purpose called `forwardRef`, and we use it like this: ~~~js export const Button = React.forwardRef(function Button(p, ref) { ... } ~~~ Let me try to break this down. A [higher-order function](https://www.oreilly.com/library/view/functional-programming-in/9781492048633/ch04.html) is a function that takes functions as parameters or returns a function. Here `forwardRef` takes that `Button` function that renders the component as a parameter. Thanks to `forwardRef`, development teams can pass refs to the form components in our library, which we pass along though that function parameter to our rendered implementation. The type parameters to `forwardRef` provide type safety and inference. The type of `p` is `ButtonProps`, and the `ref` will be hooked onto a `HTMLButtonElement`. In the end, it's a little complicated and a fair bit of ceremony, but the result is pretty simple--a form component that accepts a `ref` from the caller so form libraries can work with it as needed. ### Directory Structure When considering how to lay out your source code, it comes down to your team's preference, but as I posted recently: > There is a lot of commentary on how we should lay out source code in React. If you take two "things" (functions, classes, #TypeScript interfaces, etc.), the higher the frequency that changing one changes the other, the closer they should be together. What does that really mean in practice? Simple. When it comes to our component library, this means organizing code dedicated to a particular component in the same directory and even in some cases the same file. This is how we do it at a high level. ![Button component directory layout](https://www.vidyasource.com/img/blog/rcl-button.png) Our `Button.tsx` contains the `ButtonProps` interface, related types, and of course the component itself. Meanwhile, I love how Chakra UI allows us to separate theming from behavior, so the colors, spacing, font family, icon sizes, focus behavior, and other button details defined by our design system are in `ButtonTheme.ts`, a different file in the same directory. Finally, although we could keep our tests and stories (more on these later) in the same directory, we prefer organizing them in their own subdirectories. I guess I've seen too much Marie Kondo. ### TypeScript Config I come from a background in [statically and strongly typed programming languages](https://stackoverflow.com/questions/2690544/what-is-the-difference-between-a-strongly-typed-language-and-a-statically-typed) like Java and Scala. While I understand longtime JavaScript engineers balk at types, I find types make me extremely productive. As a result, our TypeScript config is very strict. In particular from our `tsconfig.json`: ~~~json { ... "compilerOptions": { ... "noUnusedParameters": true, "noImplicitReturns": true, "noFallthroughCasesInSwitch": true, "noImplicitAny": true, ... }, ... } ~~~ As for building the library for application development teams, we scope our `tsconfig.json` this way: ~~~json { ... "include": [ "src/**/*" ], "exclude": [ "**/__stories__/*", "**/__test__/*" ], ... } ~~~ All our components, stories, and tests are in the `src` directory, but we only want the components when we build the library. This is why we exclude the `__stories__` and `__test__` directories inside each component directory. ### Static Analysis and Code Formatting Like everyone else, we rely on eslint and Prettier, and we don't do anything particularly special. Still, I do want to mention a couple of things. First is `eslint-plugin-jsx-a11y`. We use [this eslint plugin](https://github.com/jsx-eslint/eslint-plugin-jsx-a11y) to automate verification of the accessibility of our component library. It checks the JSX of our components for obvious violations. This is as far as we can go with automation, but we complement `eslint-plugin-jsx-a11y` with manual auditing in Storybook I will discuss shortly. There might be something gnawing at the experienced engineers reading this. In the `tsconfig.json` above, we exclude our stories and tests because they don't belong in the build. Still, you know we should apply the same quality standards to story code and test code as we do to production code. Code is code. To do this, we [extend](https://www.typescriptlang.org/tsconfig#extends) `tsconfig.json` in a file called `tsconfig.eslint.json`, replacing the `exclude` field with an empty array, and configure `eslint` to use *that*. This tells `eslint` (and therefore Prettier) to include *everything* in the `src` folder in its analysis with identical TypeScript configuration. This means, for example, we can't cheat by using an implicit `any` in our stories or tests either. ### Builds We run our builds with [Vite](https://vitejs.dev/). That may seem counterintuitive since Vite is the build tool for [Vue](https://vuejs.org/) while our library is built with React, but Vite is actually agnostic. In fact, it amazed me how little configuration we needed. It basically just worked. Our Vite config is almost identical to the [example in the documentation](https://vitejs.dev/guide/build.html#library-mode). Just like the example, our build produces two bundle formats--`es` and `umd`--and it works fast. As you may know, TypeScript builds feature two phases, type checking and transpilation to JavaScript. Type checking by `tsc`, the TypeScript compiler, is *very* slow, so while it is very important, you should do it rarely. We only do it via the IDE in real time as we code or when we build the library for production--and break the build if type checking fails. We have a dedicated `typecheck` script in our `package.json` that looks like this: ~~~json { "scripts": { ... "typecheck": "tsc --p tsconfig.eslint.json --skipLibCheck --sourceRoot src --noEmit", ... } } ~~~ Note that we use `tsconfig.eslint.json` to typecheck everything. Meanwhile, transpiling your TypeScript source code to JavaScript is faster than type checking, but so is reading Tolstoy. Transpiling with `tsc` or Babel is still not fast. However, the transpiler [esbuild](https://esbuild.github.io/) is written in Go, a language [built for speed](https://www.vidyasource.com/blog/scala-go/), and Vite uses it under the hood. Because we are transpiling constantly to see what's happening in Storybook, it's crucial that the process be fast. Thanks to esbuild, Vite does exactly what we need. Our production build, versioned with [Semantic Versioning](https://semver.org/), includes [declaration files](https://www.typescriptlang.org/docs/handbook/declaration-files/introduction.html) for each component and an `index.d.ts` file enumerating all components. These improve DX by enabling developers' IDEs to perform fast code completion. We also provide the [theme file](https://chakra-ui.com/docs/theming/customize-theme) we use for our own components so that developers can apply the same theme to theirs. Our CI/CD pipeline publishes the library to a private NPM registry, which allows appropriately configured `npm` installations on developer machines to fetch the library with a conventional `npm install`. The `package.json` file accompanying the library contains all the peer dependencies they will need to use the library so `npm` can grab them, and for convenience it also contains the version of the design system it is built with for developers to track. It also contains configurations to define which files to package in the library and how consumers can import modules: ~~~json { ... "files": [ "dist" ], "types": "./dist/index.d.ts", "main": "./dist/components.umd.js", "module": "./dist/components.es.js", "exports": { ".": { "import": "./dist/components.es.js", "require": "./dist/components.umd.js" } } ... } ~~~ One last thing to note about the build. Although Vite of course provides minifying and other production readiness capabilities, we don't use them. We bundle the component library completely "raw." We find this helps developers debug their applications and report bugs (in those rare cases we make mistakes) with specificity. When they run their own builds, their tooling will apply minifying, tree shaking, and all other production processing to all their code and dependencies including the component library. ## Testing As I mentioned before, we limit the functionality of our components to the bare minimum necessary to add value. Still, components are code, and our consumers have expectations of our code. This means we need to test our components as much as we can and where it makes sense. Testing is a controversial topic. On Tech Twitter, engineers are more than happy to let you know why you are wrong to test your code in a different way than they do. I can only describe what works for us and why we think so while also stipulating that our methods are subject to change as we get better at this. Our approach is heavily inspired by this [Storybook blog post](https://storybook.js.org/blog/how-to-actually-test-uis/). In it, [Varun Cachar](https://twitter.com/winkerVSbecks) describes different types of testing, when each is appropriate, and which tools make sense for which types based on the experiences of several large-scale engineering teams. ### Storybook Storybook is crucial to the development and testing of the component library for us, and it's indispensable documentation for our users. During development, we use it in a couple of ways. If the component is simple, then it's nice to have your code and Storybook side by side and watch your changes render as you make them with hot reload. On the other hand, when we aren't clear on what the API for a component should be, it's nice to write a few [stories](https://storybook.js.org/docs/react/get-started/whats-a-story) to work out the DX for it. Experienced engineers might recognize this approach as analogous to [Test-Driven Development (TDD)](https://www.agilealliance.org/glossary/tdd/). We apply our design system custom theme in Chakra UI to every story in `preview.jsx`: ~~~js export const decorators = [Story => {Story()}] ~~~ During testing, we also use Storybook in multiple ways. For example, because we take a mobile first approach to our components, which matters for [organisms](https://bradfrost.com/blog/post/atomic-web-design/#organisms) in particular like modals, we configure custom breakpoints like this in `preview.jsx`: ~~~js export const parameters = { viewport: { viewports: { xs: { name: "XS", styles: { height: "568px", width: "320px", }, type: "mobile", }, sm: { name: "SM", styles: { height: "896px", width: "480px", }, type: "mobile", }, md: {...}, lg: {...}, xl: {...}, defaultViewport: "xs", }, } ~~~ I mentioned a CI/CD pipeline that builds the library and publishes it to a private registry. It turns out the pipeline also publishes our component Storybook to an [Nginx container](https://hub.docker.com/_/nginx) so that the UX team can conduct visual testing on the components, and the ability to toggle among viewport sizes is extremely helpful. It's also helpful for development teams who use our components to interact with them. Thanks to [Storybook Controls](https://storybook.js.org/docs/react/essentials/controls), they can configure components themselves to see what happens. Thanks to [Storybook Docs](https://storybook.js.org/addons/@storybook/addon-docs), they can see the code and API props that generate each story. So Storybook provides a profound documentation benefit throughout the program. We also use Storybook for [composition testing](https://storybook.js.org/blog/how-to-actually-test-uis/) occasionally though not as often as the Storybook team may prefer. For example, we have stories that demonstrate how to integrate our form components with React Hook Form, and this exposed issues we had with our `ref`s. Generally though, we don't do a lot of composition testing until we need to [reproduce a scenario to fix a bug](https://www.vidyasource.com/blog/code-coverage-is-killing-you) (and prove we've fixed it eventually). We make heavy use of [storybook-addon-a11y](https://storybook.js.org/addons/@storybook/addon-a11y) to test for accessibility. As you can see from another post by [Varun Cachar](https://twitter.com/winkerVSbecks), who is definitely earning his paycheck, [Storybook offers a lot of features for accessibility testing](https://storybook.js.org/blog/accessibility-testing-with-storybook/). We make use of all of them. As I mentioned before, even though we do our best with `jsx-a11y` in the build and Storybook visually to test for accessibility, it is still incumbent upon teams to add [@axe-core/react](https://www.npmjs.com/package/@axe-core/react) to *their* builds and perform their own visual tests in order to feel as confident as we can that we are providing the best possible experience to all our users. Finally, while Storybook has been invaluable for us and I recommend it strongly, I would be remiss if I didn't mention some gotchas. Storybook uses a lot of the same libraries we all use for theming, Markdown, and other things. When there are library conflicts between your version and theirs, bad things happen. For example, we got hit with the same conflict on [Emotion](https://emotion.sh/docs/introduction) as this [issue on GitHub](https://github.com/storybookjs/storybook/issues/15879). To its credit, the Storybook team releases frequently. If nothing else, make sure you use identical versions of Storybook and all its addons and that you upgrade as soon as possible when updates are available. Storybook is also well aware of the "DivOps" revolution in JavaScript build tooling [and is positioning itself accordingly](https://storybook.js.org/blog/storybook-for-webpack-5/). This is exciting since Webpack had a good run but feels more and more like the past, and we wanted to use Vite with Storybook. We installed [storybook-builder-vite](https://storybook.js.org/blog/storybook-for-vite/) knowing it's experimental to see how it would work for us. Overall, it makes our Storybook builds fast just as we hoped. Still, when you consider `storybook-builder-vite` is raw, community-led by great engineers who have already given the community so much with their limited time and can't address every issue, and the general brittleness of Storybook I mentioned, your mileage may vary. Here is our Vite-related Storybook configuration in `main.js`: ~~~js module.exports = { ... core: { builder: "storybook-builder-vite" }, viteFinal: async config => { return { ...config, plugins: ..., optimizeDeps: { ...config.optimizeDeps, entries: [`${path.relative(config.root, path.resolve(__dirname, "../src"))}/**/__stories__/*.stories.@(ts|tsx)`], }, } }, } ~~~ ### React Testing Library If you have read any of my posts on testing, you know that I think our industry writ large gets testing wrong. We test some things too much. We test other things too little. We don't always know the purpose of our tests. And worst of all, because of perverse incentives, [we write tests to check a box](https://www.vidyasource.com/blog/code-coverage-is-killing-you). I mentioned earlier that it has been a priority to endow our components with as little behavior as possible. Aside from the fact simpler code is easier to maintain and understand, this approach means fewer surprises for our consumers and less for us to test. Or so I thought. Our program has a mandatory minimum of 80% code coverage for our applications, and for reasons that don't make a lot of sense to me, that also applies to the component library. In my view, only components that maintain internal state offer the complexity that demands the ceremony of formal tests beyond Storybook, but alas, I don't make the rules. React Testing Library has become the *de facto* standard for [interaction testing](https://storybook.js.org/blog/how-to-actually-test-uis/) in React, and of course we use it for our own tests. But how could we write tests as quickly as possible to limit the impact of the code coverage standard? If you have written tests in any programming language, you understand the concept of "[test fixtures](https://stackoverflow.com/questions/12071344/what-are-fixtures-in-programming)," the setup for your tests. For us, that means test fixtures are simply components configured with different props. But isn't that exactly what stories in Storybook are? Storybook offers a feature I love--the ability to import stories into tests written with React Testing Library as fixtures using [@storybook/testing-react](https://storybook.js.org/addons/@storybook/testing-react). Without it, we would need to duplicate the same code as stories in Storybook and fixtures in tests. The autocompletion is great too thanks to the TypeScript support built into `@storybook/testing-react`. One last thing I want to mention is, as you might guess given how much I have emphasized it in this post, accessibility. All of our tests in React Testing Library use `getByRole` and `findByRole` selectors. We do this because it is a way to build implicit accessibility testing into our interaction tests as [the documentation describes](https://testing-library.com/docs/queries/about#priority). After all, if we are unable to locate the component we wish to test by its ARIA role, that all but guarantees it isn't accessible. And if it isn't accessible, I don't care if it "works" because it doesn't work for everyone. Aside from all that, our tests work exactly as you would expect if you know React Testing Library. Here is an example of a simple test conveying everything I described: ~~~js ... import { DefaultMediumPrimaryButton, ... } from "../__stories__/Button.stories" test("Button primary display works", () => { const onClickMock = jest.fn() render() const button = screen.getByRole("button", { name: "Primary" }) userEvent.click(button) expect(onClickMock).toHaveBeenCalledTimes(1) }) ~~~ --- I know this is a lot, and it might have been slightly more entertaining as an audiobook. Still, I hope I conveyed the value in design systems and component libraries and the lessons we learned in project management, communications, accessibility, engineering, and testing to build something that will impact the lives of millions. I hope you can do the same...but better. Now go take a nap. You earned it. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Scala vs Go: Who Wore It Better?" description: "Scala vs Go compared on error handling, collections, and absent values. Which one fits your project?" canonical: https://www.vidyasource.com/blog/scala-go/ type: article published: 2021-09-01 author: "Neil Chaudhuri" tags: ["Golang", "Play Framework", "SBT", "Git", "Programming", "Scala", "Go", "Software Engineering", "DevOps", "Functional Programming", "Microservices", "REST", "Agile", "TypeScript", "Elm", "Kotlin"] image: https://www.vidyasource.com/img/blog/scala-go.png --- # Scala vs Go: Who Wore It Better? > Scala vs Go compared on error handling, collections, and absent values. Which one fits your project? - Canonical page: https://www.vidyasource.com/blog/scala-go/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/scala-go.json - Published: 2021-09-01 - Author: Neil Chaudhuri - Topics: Golang, Play Framework, SBT, Git, Programming, Scala, Go, Software Engineering, DevOps, Functional Programming, Microservices, REST, Agile, TypeScript, Elm, Kotlin Scala and Go are two of the fastest growing leading-edge programming languages in the world. In the United States, they are also among the [most lucrative](https://adtmag.com/articles/2017/08/18/go-scala-salaries.aspx). Scala and Go are among a slew of programming languages that innovate in numerous ways to produce faster, more resilient, more secure applications for a multicore, cloud native, mobile world. The thing is Scala and Go have *very* different philosophies on what makes engineers most productive and what defines great applications. We are going to look at how Scala and Go solve five common programming tasks and how their contrasting approaches reflect their contrasting philosophies. Then you can decide for yourself which works best for your next application. This is a really long post, so here are the tasks we will consider so you can jump to the ones that interest you most: * [Absent Values](#absent-values) * [Error Handling](#error-handling) * [Collections](#collections) * [Concurrency and Parallelism](#concurrency-and-parallelism) * [Polymorphism](#polymorphism) Or you can skip straight to my [conclusion](#how-do-you-decide), which boils down to this: > Despite the strong possibility that this opinion will subject me to ritual humiliation on social media, I would use Scala (or a similarly featured language like Kotlin) for microservices or bigger but Go both to replace any bash or Python scripts that are part of my continuous delivery pipeline and to create lambda functions, which are supposed to be lightweight, fast, and focused. With that, let me introduce you to Scala and Go. ## Scala Scala is an object-oriented (OO) *and* functional programming language built for the JVM (though [Scala Native](https://scala-native.readthedocs.io/en/latest/) is in the works). It emerged from academia at the [EPFL](https://scala.epfl.ch/) in Switzerland from the mind of Academic Director Martin Odersky, who sought to prove that the two paradigms--OO for representing your domain and functional for its [mathematical guarantees for working programs](https://www.vidyasource.com/blog/business-case-for-functional-programming/)--can blend seamlessly to yield the best of both worlds. Scala also has an exquisite static type system that can provide a powerful safety net. Operationally, as you might expect from a language borne from academia, Scala tooling can be problematic and compilation can be slow--particularly if you are not yet using Scala 3, which only recently emerged and is very slowly percolating through the ecosystem (Remember the Python 2 to Python 3 transition?). But type inference, a vast standard library, and the time-tested reliability of the JVM make you very productive once you get the hang of them. Performance varies with the JVM you're running, but regardless you do have to contend with the size of compiled objects and the latency of garbage collection at runtime. When you want to experiment, you can skip the ceremony of writing a class or test and instead use a command-line REPL, an online REPL called [Scastie](https://scastie.scala-lang.org/) you can share, or an outstanding third-party command-line REPL called [Ammonite](https://ammonite.io/#Ammonite-REPL). Dependency management is achieved with SBT typically but also more general JVM build tools like Gradle and Maven. Scala became really popular with the advent of "Big Data" because [functional programming lends itself so naturally to analytics](https://www.vidyasource.com/blog/java-is-dysfunctional-with-big-data), and the learning curve for [modern LISPs like Haskell and Clojure](https://en.wikipedia.org/wiki/Lisp_(programming_language)#2000_to_present) is too high for too many. Apache Spark is built in Scala, and when it got big, Scala got big. Since then Scala has also become a popular language for other domains including [reactive web applications and microservices](https://www.reactivemanifesto.org/) with [Play Framework](https://www.playframework.com/) and [Akka](https://akka.io/) and even the front end with [Scala.js](https://www.scala-js.org/). ## Go Go had an pragmatic mission from the start, so it was built with simplicity, minimalism, and performance in mind. Robert Griesemer, Rob Pike, and Ken Thompson created Go at Google as an [alternative to C++](https://talks.golang.org/2015/gophercon-goevolution.slide#4) built for modern machines. Go is compiled to native machine code; no virtual machine. Go is statically typed like Scala but is imperative and procedural. It's not a functional language *per se*, but [functions are first-class](https://golangbot.com/first-class-functions/). Where it truly shines is in its concurrency primitives: [channels and goroutines](https://medium.com/rungo/anatomy-of-channels-in-go-concurrency-in-go-1ec336086adb). Together, they enable you to leverage the full power of your multicore machine. Operationally, Go is really fast to compile and run. It too has a vast standard library for common tasks like [REST calls](https://golang.org/pkg/net/http/) and [JSON de/serialization](https://golang.org/pkg/encoding/json/). When you want to experiment, you can use the [Go Playground](https://play.golang.org/) or a scratch file in your IDE, but there is no command-line REPL. Unlike Scala, Go has by design a very lean feature set and simple constructs. This makes it relatively easy to learn. However, you have to add a lot of features yourself that you take for granted in other languages or explore what Go offers as [potential workarounds](https://golang.org/doc/faq#Why_doesnt_Go_have_feature_X). To get *really* good at advanced features of Go like concurrency and polymorphism is still a challenge. Official dependency management is nonexistent, and you may find Go's unorthodox project setup based on Git repos [takes getting used to](https://medium.com/rungo/working-in-go-workspace-3b0576e0534a). After Go took off at Google and was released to the public, it got really popular as the language of concurrency, which helped in turn to make it the language of DevOps--particularly in concert with [Kubernetes](https://rancher.com/using-kubernetes-api-go-kubecon-2017-session-recap/), which also emerged from Google. Go has expanded into other domains as well with the CMS [Hugo](https://gohugo.io/) and the microservices framework [Go kit](https://gokit.io/). ## Comparing Scala and Go Let's look at how Scala and Go handle the kinds of real-world problems you will encounter every day. *Disclaimer:* The sample code is not meant to showcase necessarily the "best" way to solve these problems--most performant, most elegant, or whatever. Instead it is meant to showcase reasonable, idiomatic solutions and, more importantly, how they reflect the design philosophies of the respective languages. Of course [you are welcome to suggest improvements](https://www.vidyasource.com/contact/) regardless. Also, the Scala code uses some of the revolutionary [Scala 3](https://docs.scala-lang.org/scala3/new-in-scala3.html) syntax and constructs. ### Absent values You have to deal with potentially absent values all the time like when database queries for single entities return no hits or when you are maintaining backwards-compatible microservices. Typically, absent values are represented with `null`. Sir Tony Hoare called his invention of `null` to represent the absence of a value his "[billion-dollar mistake](https://www.infoq.com/presentations/Null-References-The-Billion-Dollar-Mistake-Tony-Hoare)." Handling absent values isn't glamorous, but if you do it poorly, you will suffer significant productivity loses. #### Scala Although `null` exists in Scala as a JVM language, you should (almost) never interact with it directly. Instead, you should work with `Option`, a [monad](https://stackoverflow.com/questions/44965/what-is-a-monad) designed specifically for this purpose. We describe `Option` in more detail in [our tutorial](https://www.youtube.com/watch?v=rbZ6GzR8B7I), but essentially it compels you to account for the potential absence of a value at *compile* time. This avoids the costly `NullPointerException` at runtime that has sent thousands of Java developers to therapy. ~~~scala case class Student(name: String, house: String) def findStudent(key: Int): Option[Student] = { val students = Map( 1 -> Student("Harry Potter", "Hogwarts"), 3 -> Student("Draco Malfoy", "Slytherin") ) students.get(key) } findStudent(3) .map(s => s"The student's name is ${s.name}") .getOrElse("Back to Hogwarts") ~~~ In this example, the `findStudent` function returns an `Option[Student]`. If the `Option` contains a value, the client can transform it into a `String` (*i.e.* `Option[Student] => Option[String]`) with the `map` function on `Option`; otherwise, `getOrElse` handles the absent value. `Option` allows you to safely work with potentially absent values [without fear](https://marvel.fandom.com/wiki/Daredevil:_The_Man_Without_Fear_Vol_1_1). The compile-time safety makes you very productive. On the other hand, every transformation on the `Option`--via `map`, `filter`, *etc.*--produces a new value because of the functional programming bias towards [immutability](https://www.vidyasource.com/blog/business-case-for-functional-programming/). If memory is at a premium, maybe this is a concern. #### Go Go handles potentially absent values through completely different idioms. Functions can return multiple return values. When a value is present, it's idiomatic to return the value and a `bool` value (named `ok` by convention) of `true`; otherwise, it returns the ["zero value" of the return type](https://tour.golang.org/basics/12) and `false`. The client uses an imperative `if` statement to distinguish. ~~~go package main import "fmt" type Student struct { name string house string } func findStudent(key int) (Student, bool) { students := map[int]Student{ 1: Student{name: "Harry Potter", house: "Hogwarts"}, 3: Student{name: "Draco Malfoy", house: "Slytherin"} } student, ok := students[key] return student, ok } func main() { if student, ok := findStudent(3); ok { fmt.Printf("The student's name is %v\n", student.name) } else { fmt.Println("Back to Hogwarts") } } ~~~ In this example, the `findStudent` function returns a `Student` *and* a `bool`. Go allows you to initialize and test conditions in one line, and that's what we see here. If `ok` is `true`, we know we got something and act accordingly; if `ok` is `false`, we know the value is absent and handle this contingency. For those uncomfortable with monad composition and higher-order functions, Go provides a very simple alternative in keeping with its mission. On the other hand, Go engineers need the knowledge and discipline to apply these language features and idioms. There is no dedicated type like `Option` to help. Also, unlike with the composability afforded by monads like `Option`, if you need to compose multiple potentially absent values, you need to write multiple `if` conditions; that can feel verbose. The simplicity may well be worth it though. ### Error Handling This is another inglorious task that is critical to good software engineering. As programs become big and complex, proper error handling is critical to diagnose bugs and move builds to production as quickly as possible. #### Scala As you might imagine from a language that prizes on immutability and composability, Scala offers another monad, `Try`, for error handling. Analogous to `Option`, it compels you to account for a possible error at compile time rather than the absence of a value. A `Try[Double]`, for example, represents either a `Double` if all is well or an error (or more precisely an instance of `Throwable`) otherwise. ~~~scala import scala.util.{Failure, Success, Try} def squareRoot(value: Int): Try[Double] = { if (value >= 0) { Success(Math.sqrt(value)) } else { Failure(new IllegalArgumentException("Value cannot be negative")) } } val sum = for a <- squareRoot(4) b <- squareRoot(16) yield a + b sum.map(s => s"The result is $s") match { case Success(result) => result case Failure(t) => t.getMessage } ~~~ In this example, the `squareRoot` function returns `Try[Double]`, and the client does something a bit more complicated than the prior example. It calls `squareRoot` twice and uses a [for comprehension](https://docs.scala-lang.org/tour/for-comprehensions.html), which is syntactic sugar for otherwise cumbersome `map` and `flatMap` transformations, to take advantage of the composability of monads to sum the results. If both calls work out, then the result is a `Try` with the sum; if either fails, the error information is passed on. You can do similar with `Option` too. This is why monad composability in Scala is so cool. The same pattern works on very different types as long as they follow the [monad laws](https://miklos-martin.github.io/learn/fp/2016/03/10/monad-laws-for-regular-developers.html). As before though, keep in mind that you are generating new objects with each transformation. This means you're paying for the protection immutability affords you with memory. #### Go Just as `Option` and `Try` in Scala are similar, handling absent values and handling errors in Go is similar. Here again Go takes advantage of multiple return values, but rather than a value accompanied by a `bool`, idiomatic Go error handling features a value accompanied by an `Error` (named `err` by convention). If `err` has the value of `nil`, then you can work with the value because it's all good; otherwise, you handle the error and (probably) ignore the zero-value. ~~~go package main import "fmt" import "math" func squareRoot(value int) (float64, error) { if value >= 0 { return math.Sqrt(float64(value)), nil } else { return 0, fmt.Errorf("invalid input: %v", value) } } func sumRoots(v1, v2 int) (float64, error) { a, err := squareRoot(v1) if err != nil { return 0, err } b, err := squareRoot(v2) if err != nil { return 0, err } return a + b, nil } func main() { if sum, err := sumRoots(4, 16); err == nil { fmt.Printf("The sum is %v\n", sum) return } else { fmt.Println("Value cannot be negative") } } ~~~ In this example, the `sumRoots` function, which is the client of `squareRoot`, returns a value and an error. Those are saved from the first call to `squareRoot`. If there is an error, the function returns immediately; otherwise, it does the same with the second call to `squareRoot`. If the function makes it to the end, then it returns the sum and a `nil` error. You will see a similar approach in `main` with the call to `sumRoots`. There are a few interesting things to note. First, once again the code reflects Go's bias toward simplicity. Furthermore, as immutability is not a priority, variable reuse is common in Go. In this case, `err` is reused to store the error returned from `squareRoot`. It doesn't happen here, but it isn't unheard of to reuse the value variable as well once its initial value has served its purpose. Finally, we see a common pattern: * Call a function * Test `err` for `nil` * If an error is found, return * Call the next function * Test `err` for `nil` * If an error is found, return * Lather, rinse, repeat Absent Scala's function composition, it's a fair bit of boilerplate--particularly when you are calling a lot of functions that may return errors. You can take advantage of language features to make it more elegant like [this pattern utilizing interfaces and pointers from Rob Pike](https://golang.org/doc/faq#exceptions), but this is one of the ways where Go's simplicity is a bit overrated. Building custom abstractions from Go's toolkit requires creativity, and you will have to do that often when you use Go to build mature applications. Still, it is almost certainly easier to master advanced patterns in Go than it is in Scala. Finally, even though it doesn't appear in this example, `defer` [is a simple but powerful construct](https://gobyexample.com/defer) in Go that executes a call at the end of a function no matter what happens--error or otherwise. You can use `defer` to clean things up after an error in the same way you'd use `finally` in other languages. ### Collections I don't have to tell you that manipulating data from a database, a [stream](https://medium.com/stream-processing/what-is-stream-processing-1eadfca11b97), a REST request, or a host of other sources is a common task in application development. Languages that facilitate seamless transformation and aggregation of collections of data make your life a lot easier. #### Scala Scala has a vast library of collections--both immutable and mutable though immutable is preferred--each of which offers in turn a vast collection of higher-order functions that let you manipulate the collection in numerous ways. ~~~scala val words = List("Fear", "of", "a", "name", "only", "increases", "fear", "of", "the", "thing", "itself") words .groupBy(_.toLowerCase) .mapValues(_.size) ~~~ In this example, given a `List` of words, the code uses `groupBy` to convert it into a `Map` of each word (lowercased to normalize them) to a list of occurrences of each word. Finally, those lists are transformed into their sizes, and the result is a `Map` of each word to its count. There isn't much to see. This is a credit to Scala's powerful abstractions. Still, it is important to keep in mind that this code performs multiple *O(n)* traversals and with each one consumes memory to produce an entirely new collection--all but the first and last of which are immediately thrown away. #### Go Go naturally takes a far more lightweight approach to collections. You will essentially only deal with [maps](https://blog.golang.org/go-maps-in-action) and [slices](https://blog.golang.org/go-slices-usage-and-internals), which are array views that enable memory-efficient operations on the backing arrays. ~~~go package main import "fmt" import "strings" func main() { dictionary := make(map[string]int) words := []string{"Fear", "of", "a", "name", "only", "increases", "fear", "of", "the", "thing", "itself"} for _, word := range words { word = strings.ToLower(word) if count, ok := dictionary[word]; ok { dictionary[word] = count + 1 } else { dictionary[word] = 1 } } fmt.Printf("The counts are %v\n", dictionary) } ~~~ In this example, the code uses `make` both to create an empty map (with values defaulting to the zero value, in this case literally 0) to hold the word counts and to create a slice containing the words. It simply iterates over the slice and builds the word count using the `ok` idiom you saw before to check if there is an existing entry for the word in the map. The code is more verbose than its Scala counterpart, but it is significantly more efficient in time and space. There is a single *O(n)* traversal with constant-time lookups in the map, and we maintain only two collections the entire time. The efficiency of Go collections and the simplicity of using them are among the best reasons to use Go in a project. ### Concurrency and Parallelism Modern software applications have a lot more to do in a lot less time, so it's important for your code to take advantage of every bit of power your machines have. Modern applications demand modern languages that enable you to leverage every core through abstractions that strike the right balance of power and intuitiveness. Perhaps most importantly, they need to provide mechanisms for handling errors because concurrent/parallel programming is notoriously hard to debug. It's a challenge to reproduce the conditions that generated the bug in the first place. By the way, as Rob Pike has taught us, [concurrency is not parallelism](https://www.youtube.com/watch?v=cN_DpYBzKso). Long story short, concurrency is about decomposing a problem into its components; each component might then run in parallel depending on available resources. Concurrency *manages* a lot of things at once while *parallelism* does a lot of things at once. Regardless, doing concurrency and parallelism well is a hard problem. Whatever path you choose, you need to understand the nature of your tasks to achieve peak performance. Are they IO- or CPU-intensive? Are they intrinsically parallel? #### Scala As a core language in [reactive programming](https://www.reactivemanifesto.org/), Scala takes concurrency and parallelism very seriously. It offers `Future` as its [core primitive](https://docs.scala-lang.org/overviews/core/futures.html) to facilitate concurrency and parallelism--in concert with an `ExecutionContext`, which is basically a [thread pool](https://www.playframework.com/documentation/2.7.x/ThreadPools). `Future` abstracts the threads away from you, which is nice, but its association with `ExecutionContext` can lead to some [complexity](https://stackoverflow.com/questions/27454798/is-future-in-scala-a-monad) in understanding [how they work](http://www.beyondthelines.net/computing/scala-future-and-execution-context/). This is why notable [third-party libraries](https://alvinalexander.com/scala/differences-scalaz-task-scala-future-referential-lazy) offer their own concurrency primitives. Still, `Future` at least approximates a monad, and that means we can *mostly* adhere to the familiar patterns we saw with `Option` and `Try`. At a low level, each asynchronous call delegates to a different thread. This helps scale your application, but it's also a heavyweight operation as the operating system needs to schedule threads against physical processors and manage expensive context switching. The result is that for some problems parallelism with `Future` in Scala may potentially consume a lot of resources for merely a modest increase in performance--or even slow you down. When performance is a major concern, you need to configure your `ExecutionContext` smartly according to the capability of your machine(s) and the nature of your tasks. Bottom line? Reactive programming doesn't necessarily make your applications faster (except maybe in those cases where you can do expensive work in parallel), but it usually allows them to be more resilient. Reactive applications scale under load with limited threads and memory especially when you have latency from inconsistent network IO like database and REST calls. ~~~scala import akka.actor.ActorSystem import akka.stream.ActorMaterializer import play.api.libs.ws._ import play.api.libs.ws.ahc._ import scala.concurrent.Future import scala.concurrent.ExecutionContext.Implicits._ import play.api.libs.json._ implicit val system = ActorSystem() system.registerOnTermination { System.exit(0) } implicit val materializer = ActorMaterializer() val ws = StandaloneAhcWSClient() def getUuid: Future[String] = { ws.url("https://httpbin.org/uuid") .withHttpHeaders("Accept" -> "application/json") .get .map { response => (response.body[JsValue] \ "uuid").as[String] } } val uuid1: Future[String] = getUuid val uuid2: Future[String] = getUuid val uuids = for u1 <- uuid1 u2 <- uuid2 yield (u1, u2) uuids .foreach { case (u1, u2) => println(s"UUIDs: $u1, $u2") } .andThen { case _ => ws.close() } .andThen { case _ => system.terminate() } .recover { case t: Throwable => println(s"There was a problem: $t") } ~~~ In this example, the code uses [Play WS Standalone](https://github.com/playframework/play-ws) as a REST client to fetch JSON containing a [UUID](https://en.wikipedia.org/wiki/Universally_unique_identifier). Play WS has an asynchronous, non-blocking API based on `Future`, so you need to provide an `ExecutionContext` via [Akka](https://doc.akka.io/docs/akka/2.5/stream/). That's all the boilerplate at the beginning of this example. Sometimes it will be done for you as when you use Play WS in the context of [Play Framework](https://www.playframework.com/). Nonetheless, you should be aware it has to happen somewhere. In the `getUuid` function, the `get` call to Play WS makes an asynchronous, non-blocking HTTP GET request and returns a `Future` containing the response. You need to be careful and make sure you only work directly with the `Future` itself lest you lose the eventual result--the response or an error. Otherwise anything you do next outside the realm of the asynchronous call will execute after the request is dispatched but completely independently from when the response returns. That's a common mistake for engineers new to `Future`. This is why everything that happens after the call in `getUuid` is dispatched is a method call on the `Future` object that comes back. Like a typical monad, `Future` offers `map` to enable clients to transform results, and `getUuid` does exactly that by parsing the JSON response into the UUID string and returning a `Future[String]`. If the REST call returned an error, it remains preserved in the `Future.` The client of `getUuid` requires two calls to complete before moving forward, so it calls the function twice so the independent fetches can run in parallel and composes them as we saw with `Try` via `for` comprehension to produce a `Future[Tuple2[String, String]]`. Finally, the code calls `map` again to transform the tuple of strings into a printed result. If there is an error any step of the way, we handle it with `recover`. `Future` is a really nice concurrency primitive that offers the convenience of a monad, but it has its pitfalls. As mentioned, you need to supply a finely tuned `ExecutionContext`, and everything you do once you make an asynchronous call better happen in the context of the `Future` that returns. Otherwise, you will find very confusing results like silent failures. The [amorphous referential transparency](https://www.reddit.com/r/scala/comments/3zofjl/why_is_future_totally_unusable/) of `Future` can confuse you too. If the code made its calls to `getUuid` inside the `for` comprehension, they would have been sequential and not parallel. The result is likely the same, which *feels* [referentially transparent](https://alvinalexander.com/scala/how-to-create-scala-methods-no-side-effects-pure-functions#referential-transparency), but the fact where you make the call impacts the desired parallelism definitely doesn't. `Future` also consumes a lot of system resources because it transacts in full threads that have to managed by the operating system. It's important to profile your application to understand what's going on because there will almost be certainly occasions where things don't perform as you expect. You will need to diagnose if it is a code or resource problem. #### Go Concurrency and parallelism lie at the heart of Go's mission as well, but of course it takes a totally different approach with primitives called [goroutines](https://tour.golang.org/concurrency/1) and [channels](https://gobyexample.com/channels). The philosophy here is to break down your tasks into independent functions, and spin them off into goroutines. The key thing to remember is [goroutines are not threads](https://golang.org/doc/faq#goroutines); they are lightweight pieces of memory tracking thread usage. As a result, you can spin off literally hundreds of thousands of goroutines and use only a little memory on the stack while the Go scheduler, not you, uses clever algorithms to manage them in concert with the operating system and its actual threads. Goroutines use a paradigm called [communicating sequential processes](https://en.wikipedia.org/wiki/Communicating_sequential_processes) (CSP) developed by Sir Tony Hoare, who despite being the `null` guy is a pillar of computer science. CSP is a message-passing model. Goroutines pass their data over channels rather than synchronize data, which slows things down. If you are familiar with messaging patterns like [Gregor Hohpe's Enterprise Integration Patterns](https://www.enterpriseintegrationpatterns.com/), then you understand the power of this approach to manipulate message data as needed to produce the results you want--and at scale thanks to Go. ~~~go package main import ( "encoding/json" "fmt" "net/http" "strings" ) type Result struct { uuid string err error } func getUuid(rc chan<- Result) { r, err := http.Get("https://httpbin.org/uuid") if err != nil { rc <- Result{"", err} return } response := make(map[string]interface{}) err = json.NewDecoder(r.Body).Decode(&response) if err != nil { rc <- Result{"", err} return } rc <- Result{response["uuid"].(string), nil} } func main() { n := 2 rc := make(chan Result, n) uuids := make([]string, 0, n) for i := 0; i < n; i++ { go getUuid(rc) } for i := 0; i < n; i++ { r := <-rc if r.err != nil { fmt.Printf("There was a problem: %v\n", r.err) return } else { uuids = append(uuids, r.uuid) } } fmt.Printf("UUIDs: %v\n", strings.Join(uuids, ", ")) } ~~~ In this example, the code defines a type called `Result` containing a UUID and an error to hold the result of a concurrent computation. Only one of those should be populated with a meaningful value, but there is no simple way to enforce that. Meanwhile, the `main` function creates a [buffered channel](https://gobyexample.com/channel-buffering) for communicating `Result` values and spins off two goroutines for two concurrent `getUuid` calls, which take the channel as a write-only parameter. That *is* enforceable. The `getUuid` function makes the REST call and unmarshals the JSON response. If an error occurs at either step, the code creates a `Result` with the respective error (and the zero value, the empty string, for the UUID) and publishes it to the channel. If all goes well, the code creates a `Result` with the UUID (and the zero value, `nil`, for the error) and publishes it to the channel instead. The `main` function knows in this case how many `Results` to expect from the channel and consumes the right amount of data from the channel. It bails at the first error it finds, or it accumulates the data into a slice of UUIDs. Goroutines are lightweight and flexible--and straightforward if you have a good grasp on messaging. Still, there are subtleties that can make them more complicated than we might expect from Go. If you don't understand the difference between buffered and unbuffered channels, you may find [curious results](https://stackoverflow.com/questions/18660533/why-using-unbuffered-channel-in-the-same-goroutine-gives-a-deadlock). Error handling is also tricky because patterns aren't obvious. In this case we encapsulated success and error cases in a single struct, but some advise using [dedicated error channels](https://stackoverflow.com/a/42890750/1347281). This example is also as simple as it gets: You have one channel, and you know how many computations will transpire. In more complicated cases you will need to utilize other Go functionality from the [sync](https://golang.org/pkg/sync/) package like `Mutex`, `WaitGroup`, `Pool`. You might even need the [experimental](https://rodaine.com/2017/05/x-files-intro/) `ErrGroup`. Finally, you need to be very careful with [pointers](https://tour.golang.org/moretypes/1) and mutability generally as always in concurrent programming. They save memory but you need to govern access carefully. Concurrency and parallelism are always hard. You have to decide which language offers primitives that comport with your mental model of how things should work. ### Polymorphism You know polymorphism. Far beyond trite `Animal`-`Dog`-`Cat` examples, the business value of polymorphism is to leverage abstractions to limit changes to your code even as the functionality of your application grows. By defining new behavior leveraged through old abstractions, you can build software efficiently, and you don't have to work weekends when your client demands new features immediately. #### Scala As an OO language, Scala offers the familiar polymorphism that developers in Java, Ruby, and similar languages have loved for years, but because it is a functional language with a rich type system, it also offers [typeclass polymorphism](https://medium.com/@sinisalouc/ad-hoc-polymorphism-and-type-classes-442ae22e5342), which enables completely unrelated types to exhibit polymorphic behavior. You can think of it as functional programming's take on the [Open-Closed Principle](https://stackify.com/solid-design-open-closed-principle/) from OO. Perhaps most striking of all in comparison to Go, Scala offers parametric polymorphism--what the kids call "generics." When [Java 5 introduced generics](https://docs.oracle.com/javase/tutorial/extra/generics/index.html), it was revolutionary, and Scala benefits as well. ~~~scala // Runtime polymorphism trait Closeable: def name: String def close: String case class Connection(override val name: String, database: String) extends Closeable: override def close: String = s"Closing connection $name" case class File(override val name: String, opener: String) extends Closeable: override def close: String = s"Closing file $name" def showClosing(c: Closeable): String = c.close val w = Connection("MyConnection", "PostgreSQL") val f = File("file.txt", "TextEdit") showClosing(w) showClosing(f) // Parameteric polymorphism (generics) with List[T] val numbers = List(1, 2, 3) val firstNumber: Int = numbers.head val strings = List("1", "2", "3") val firstString: String = strings.head def use[T <: File](t: T) = s"Using ${t.name}" use(f) // Typeclass polymorphism case class Complex(real: Double, imaginary: Double) object Complex: given Ordering[Complex] with def compare(a: Complex, b: Complex): Int = a.real.compare(b.real) List(Complex(3, 5), Complex(10, 2), Complex(1, 6)).sorted ~~~ In this example, we really see three examples of polymorphism in Scala. The first is the kind of straightforward [runtime polymorphism](https://stackoverflow.com/questions/28961957/example-of-runtime-polymorphism-in-java) familiar to Java developers--except with [Scala traits rather than Java interfaces](https://stackoverflow.com/questions/16410298/what-are-the-differences-and-similarties-between-scala-traits-vs-java-8-interfa). `Connection` and `File` are marked as instances of `Closeable`, and the `showClosing` function accepts a `Closeable` parameter. Therefore either `Connection` or `File` is suitable to pass to `showClosing` and the result of the function is dynamically and polymorphically resolved. The second example of polymorphism is also familiar to Java developers. It's generics. In Scala, you never have just `List`; you have `List[T]`, where `T` is a type parameter indicating the type of item in the `List`. Scala's type inference deduces that `numbers` is an instance of `List[Int]`; `strings` is an instance of `List[String]`. The `head` method returns the first element of a `List`, which is a `T`. `T` is `Int` with `numbers` and `String` with `strings`. Finally, just for demonstration purposes, the code defines a function `use` that's not only type parameterized but type bounded. It only accepts a type that's `File` or any subclass of `File`. If you try passing a `Connection` to it, the code won't compile. The last example is what I consider the coolest and most powerful form of polymorphism in Scala--typeclass polymorphism. `List` has a `sorted` method as you'd hope, but it requires an [implicit ordering](https://www.scala-lang.org/api/current/scala/collection/immutable/List.html#sorted[B%3E:A](implicitord:scala.math.Ordering[B]):Repr). In other words, you have to tell `List[T]` how to sort its elements by providing a function that [lifts](https://stackoverflow.com/questions/17965059/what-is-lifting-in-scala) `T` into an `Ordering[T]`. This makes perfect sense, and it is enforced by Scala's type system. The code defines a type called `Complex` and a way to lift `Complex` to `Ordering[Complex]` that defines how instances of `Complex` should be sorted. Without this, `List[Complex]` wouldn't know how to sort its contents, and that last line wouldn't compile. This means you can make any `T` sortable by defining an `Ordering[T]` typeclass. More broadly, it means you can use typeclasses to add polymorphic behavior to completely unrelated types, which includes types you don't control like legacy types and/or types found in imported dependencies. Polymorphism in Scala is powerful and flexible because of its sophisticated type system. It allows you not only to extend functionality in clever ways but also to constrain the solution space. In other words, you can limit the number of ways a problem can be solved, which makes it harder to write bugs. #### Go Go is not an OO language. Structs have no capacity for inheritance by design--only composition via "[embedding](https://golang.org/doc/effective_go.html#embedding)". Go is not quite functional either. However, polymorphic behavior is not only possible but a fundamental part of the power of Go via [structural typing](https://en.wikipedia.org/wiki/Structural_type_system). You can take advantage by doing two things. First, you write [interfaces](https://gobyexample.com/interfaces), which as usual define the method signatures for a set of API calls. You can also compose interfaces via embedding. Second, you can endow any type--an existing Go type like `float64` or your own custom structs--with behavior by defining functions and assigning them to the type. When you do this, the type is called a "[receiver](https://tour.golang.org/methods/8)", and if the receiver has been assigned all the functions associated with a given interface, it is an implicit instance of that interface. You can then pass the type to any function expecting an instance of that interface, and it's resolved at compile time, which makes you more productive in stark contrast to the runtime resolution of [duck typing](https://en.wikipedia.org/wiki/Duck_typing) in dynamic languages like Python. ~~~go package main import ( "fmt" ) type Named struct { name string } type Connection struct { Named database string } type Closeable interface { close() string } func (c Connection) close() string { return fmt.Sprintf("Closing connection %v", c.name) } type File struct { Named opener string } func (f File) close() string { return fmt.Sprintf("Closing %v", f.name) } func showClosing(c Closeable) string { return c.close() } func main() { connection := Connection{Named{name: "MyConnection"}, "PostgreSQL"} file := File{Named{name: "file.txt"}, "TextEdit"} fmt.Println(showClosing(connection)) fmt.Println(showClosing(file)) } ~~~ In this example, interface `Named` is embedded in `Connection` and `File`. Note that this does not denote any relationship among them, but there is a tight coupling similar to inheritance in that any changes to `Named` are reflected wherever it is embedded. Interface `Closeable` is defined with a single function `close` with no parameters and returning a `string`. Any type with an identical function is resolved at compile time as an instance of `Closeable`, and lucky for us, `Connection` and `File` qualify as they are both receivers of a `close` function with the right signature. As a result, `showClosing` works just fine when called on both kinds of structs. That's the extent of Go's polymorphism. Clearly it is not remotely as extensive as Scala's, but it promotes two important values--composition over inheritance and abstraction over implementation. Engineers coming from traditional OO backgrounds may find Go's polymorphism takes a little getting used to, but with some creativity you will find it [quite powerful](https://talks.golang.org/2015/json.slide#1). There is no question, however, that the absence of generics can be quite jarring for more complex applications. It's such a powerful and pervasive idiom that even front end languages like [TypeScript](https://www.typescriptlang.org/docs/handbook/generics.html) and [Elm](https://elmprogramming.com/type-system.html) have it. As it happens, there has been such demand for generics from the Go community that [the maintainers have begun considering it](https://go.googlesource.com/proposal/+/master/design/go2draft-generics-overview.md). As the debate rages on whether the benefits outweigh the costs--potentially the speed and simplicity fundamental to Go's mission--just recognize you won't have the benefit of generics for a while. # How Do You Decide? Scala and Go are two great languages with fundamentally different philosophies that offer distinct advantages and disadvantages. I've tried to lay those out as simply as I can so you can extrapolate which might be best suited to your situation. Having worked on complex applications with both languages, I can tell you Scala takes longer to grasp, and you will have a harder time finding Scala engineers--especially in the United States and especially if you require onsite work. But if you have senior staff who can mentor novices and who can build abstractions that utilize functional programming's strengths and the exquisite type system to constrain wayward novices, you will find the team growing very productive very quickly. Day-to-day tasks like compilation and continuous delivery are slower, and you will often find yourself exploring Scala's rich open-source community to enhance development. Meanwhile, anyone can learn Go. The constructs are simple and lightweight, and compiling and executing are just so fast. It's amazing. Mastering the more advanced concepts of Go, however, demands effort. You will also find yourself reinventing the wheel from other languages often--like writing your own `filter` function, which is easy enough but is more plumbing than directly related to your business domain--and building creative workarounds for the limitations of Go by exploiting the powerful features Go *does* offer. Dependency management and error handling could be better, and the absence of generics can be rather painful when dealing with unknown schemas like [unmarshaling dynamic JSON](https://stackoverflow.com/questions/28877512/taking-a-json-string-unmarshaling-it-into-a-mapstringinterface-editing-an). So it all depends on if the priorities of your application and project align with the priorities of the language and ecosystem you choose. Despite the strong possibility that this opinion will subject me to ritual humiliation on social media, I would use Scala (or a similarly featured language like Kotlin for microservices or bigger but Go both to replace any [bash or Python scripts](https://www.quora.com/Where-do-we-use-Python-or-shell-scripts-in-the-DevOps-project-life-cycle) that are part of my continuous delivery pipeline and to create [lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html), which are supposed to be lightweight, fast, and focused. This means you may not necessarily have to choose because nontrivial cloud native architectures will often blend both microservices and lambdas, and you should *always* have a continuous delivery pipeline. I hope this helps. If you read the whole thing, you deserve a nap. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Code Coverage Is Killing You" description: "Code coverage is intuitive but dangerous. There are quality metrics that are so much better." canonical: https://www.vidyasource.com/blog/code-coverage-is-killing-you/ type: article published: 2021-08-16 author: "Neil Chaudhuri" tags: ["Scrum", "Kanban", "Selenium", "Testing", "Programming", "Architecture", "Software Engineering", "Agile", "Continuous Delivery", "Continuous Integration"] image: https://www.vidyasource.com/img/blog/michael-scott.jpeg --- # Code Coverage Is Killing You > Code coverage is intuitive but dangerous. There are quality metrics that are so much better. - Canonical page: https://www.vidyasource.com/blog/code-coverage-is-killing-you/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/code-coverage-is-killing-you.json - Published: 2021-08-16 - Author: Neil Chaudhuri - Topics: Scrum, Kanban, Selenium, Testing, Programming, Architecture, Software Engineering, Agile, Continuous Delivery, Continuous Integration If you are a software engineer or run software projects, [code coverage](https://stackoverflow.com/questions/195008/what-is-code-coverage-and-how-do-you-measure-it) is probably very important to you. It's intuitive. Of course more tests produce better software! It's easy to calculate. Tools, automation, and stunning charts to impress the people who pay for the occasional pizza are all readily available. The problem is code coverage is killing you. Don't get me wrong. You deserve credit for your agile commitment to quality and your investment in continuous integration and continuous delivery. But why does just about everything out there say 100% code coverage is [at best unrealistic and at worst dangerous](https://softwareengineering.stackexchange.com/questions/1380/how-much-code-coverage-is-enough)? How can achieving a perfect score on a great metric be a bad thing? But OK. What's the optimal number then? There are a lot of heuristics, but [no one really knows](https://stackoverflow.com/questions/90002/what-is-a-reasonable-code-coverage-for-unit-tests-and-why). All of this uncertainty over an apparent no-brainer should give you pause, but it is important to understand the main reasons why code coverage is so flawed. ### Code coverage assumes all your code is equally vulnerable Let's say your code includes a credit card validator. There is an infinite number of ways a string can fail credit card validation, but you only need a few tests to achieve 100% code coverage for it. Meanwhile other code may have far fewer failure scenarios. Ideally, that credit card validator should have hundreds of tests associated with it while other code has far fewer. Code coverage treats them all the same and only credits you for a small fraction of the tests necessary for your most vulnerable code, so you probably won't write any more than you have to. I believe that's what the kids call "perverse incentives." ### Code coverage assumes all your code is equally valuable Let's say your application also offers payment via money order. Almost no one will use that option; the vast majority of customers will pay by credit card. Code coverage doesn't know the difference, so it considers credit card tests and money order tests equally important. That's bad. A bug with credit card payments will cost you orders of magnitude more profit than a bug with money orders, but code coverage will never account for that. ### Code coverage demands ceremony that may not be necessary Let's not lose sight of the goal, which is to gain as much confidence as possible that your code works. Certainly that can come from tests, but it does not have to. For example, if you are building a component library in React that will be consumed by application developers, it could very well be [Storybook](https://storybook.js.org/) is all you need. After all, a button in your component library will have no functionality on its own. Its functionality will come from the `onClick` event handler passed as a prop by the library consumer. All you care about is that clicking the button does something. It's a waste of time to write a test for something so trivial just to check a coverage box when Storybook gives you that for basically free. Of course, if you insist, you can also [use Storybook stories as fixtures for your tests](https://storybook.js.org/docs/react/workflows/unit-testing). ### Code coverage tells you how much but not how well This could be the worst of all. You are investing a lot of budget and schedule in building a test suite that should raise the quality of your software, lower costs, and improve customer satisfaction. But while code coverage tells you how much you tested, it doesn't tell you how *well* you tested. You can write really poor tests (*e.g.* tests that don't offer any challenging scenarios, tests that are flaky or slow, *etc.*) that yield 100% coverage. In the end, those tests might actually be *counterproductive* because the investment could have gone elsewhere and yielded some value. Frankly, none of these concerns are debatable. The only real argument for code coverage is that for all its flaws it is still the best we can do. Obviously I cannot speak to your specific project, but my extensive experience suggests that your best case scenario--if you have a team of talented, disciplined engineers who have faced no deadline pressures to get features out fast--is that your dedication to high code coverage has improved quality but not nearly enough given how much you've spent trying. Instead, what's most likely is all your spending has produced mostly useless tests that exist to check a box rather than to improve quality, and your code is only barely better than with no tests at all. The good news is we can do a lot better for a lot cheaper. ## Percentage Metrics Experience has shown me there are a lot better metrics than code coverage. Here is a good list of percentage metrics inspired by [Kostis Kapelonis](http://blog.codepipes.com/testing/software-testing-antipatterns.html#anti-pattern-6---paying-excessive-attention-to-test-coverage). ***Percentage of Bugs Reproduced By Tests (Target: 100%)***. This is the best metric. Every bug reported by testers or users should have at least one test associated with it. ***Percentage of Tests That Change (Target: 0%)***. Too often tests are coupled to implementation, so updates to your implementation details--a new technology, updated algorithm, etc--lead to laborious updates to tests. That should stop. Shifting away from conventional, scenario-based testing to [property-based testing](https://www.vidyasource.com/blog/business-case-for-functional-programming/) where possible can help. ***Percentage of Consistent Tests (Target: 100%)***. Have you ever seen the same tests pass some days and fail others? That's not consistent, and engineers typically respond by disabling or commenting out offending tests. Instead, rewrite these tests to be deterministic unit tests, or delete them entirely. ***Percentage of Developers Writing Tests (Target: 100%)***. Everyone who writes code should write tests, but it is important not to take this in a punitive direction. The purpose is not to uncover slackers; it's to grow a culture where all developers recognize test code deserves the same care as production code and to expose the entire team to the entire codebase to promote [collective code ownership](https://martinfowler.com/bliki/CodeOwnership.html). If command-and-control types do try to use this metric to identify what they consider "weak performers," then get rid of it. It does more harm than good. ## Trending Metrics There are also trends to watch. ***Low Code Quality in Tests***. Tests are code. Your test code should be a first-class citizen subject to the same quality checks as production code--limited duplication, reusable functions, design patterns where useful, *etc.* ***Time to Write Tests***. Maybe the quality of your tests is too low. Maybe the design is poor with too many dependencies to mock. Maybe it's hard to generate test data or scenarios. Taking too long to write tests will manifest in diminishing velocity and more bugs. Better design and again [property-based testing](https://www.vidyasource.com/blog/the-business-case-for-functional-programming/) can help. ***Time to Run Tests***. One of the core tenets of agile development is rapid feedback, which is impossible if your tests take forever. This might be controversial, but I would recommend favoring unit tests and functional tests over integration tests. Unit tests are fast (when written properly). Functional tests give you the most accurate view on quality, and modern tools like [Cypress](https://www.cypress.io/) and [Playwright](https://playwright.dev/) can overcome the slowness and flakiness of older tools like Selenium. You can also get a boost from your [tooling](https://engineering.linkedin.com/blog/2018/07/how-we-improved-build-time-by-400-percent). Keep these trend lines as low and as level as possible. The best part about all of these metrics is that they are easily derived from your engineering tools. Nothing special is required of you. Of course you can always take the initiative to do some clever things specific to your domain like identifying particularly vulnerable and/or valuable parts of the codebase and ensuring there is a high level of coverage for that specific region of surface area. In the end you should be able to look at every test in your code base and recognize how it gives you confidence that your code will work in production and that no one will be working weekends. The siren song of code coverage is intoxicating, but you can do so much better. --- Vidya is a certified small business in Northern Virginia that modernizes legacy systems, builds enterprise AI, and designs cloud and data architecture for commercial companies and federal agencies. Vidya documents its delivered engagements in case studies and publishes courses, tutorials, and articles for engineers and the people who lead them. The company name is Vidya. Its website is vidyasource.com. Start at https://www.vidyasource.com/llms.txt for the index of everything Vidya publishes, or https://www.vidyasource.com/contact/ to talk to Vidya. --- title: "Dark Mode in Next.js using Tailwind CSS and React Hooks" description: "Use the power of Tailwind CSS and React Hooks to build Dark Mode users can control into your Next.js site." canonical: https://www.vidyasource.com/blog/dark-mode-nextjs-tailwindcss-react-hooks/ type: article published: 2021-08-02 author: "Neil Chaudhuri" tags: ["NextJS", "Tailwind CSS", "JavaScript", "TypeScript", "React", "Mobile", "Accessibility", "Open Source"] image: https://www.vidyasource.com/img/blog/moon.jpg --- # Dark Mode in Next.js using Tailwind CSS and React Hooks > Use the power of Tailwind CSS and React Hooks to build Dark Mode users can control into your Next.js site. - Canonical page: https://www.vidyasource.com/blog/dark-mode-nextjs-tailwindcss-react-hooks/ - Structured data (JSON-LD): https://www.vidyasource.com/blog/dark-mode-nextjs-tailwindcss-react-hooks.json - Published: 2021-08-02 - Author: Neil Chaudhuri - Topics: NextJS, Tailwind CSS, JavaScript, TypeScript, React, Mobile, Accessibility, Open Source It's quite possible that while waiting for the ads on Hulu to end you stumbled upon the [option to set your phone's theme to Dark Mode](https://www.theverge.com/2019/3/22/18270975/how-to-dark-mode-iphone-android-mac-windows-xbox-ps4-nintendo-switch). Dark Mode is becoming a staple of user interfaces on the web and mobile devices for [several reasons](https://www.forbes.com/uk/advisor/mobile-phones/what-is-dark-mode-and-should-you-be-using-it/)-- primarily to ease the strain on your eyes and to reduce battery consumption. At Vidya we pride ourselves on embracing emerging technologies and helping our clients leverage them to realize their potential. When it came time to give our website a fresh new look, we figured adding a toggle-able Dark Mode option would be consistent with that mission. This website you're reading right now supports Dark Mode. Just look at the top of the page. The site is built in [TypeScript](https://www.typescriptlang.org/) with [React](https://reactjs.org/), the most popular JavaScript library in the world, using [Next.js](https://nextjs.org/), one of the most popular React frameworks in the world and the building block for full-stack "meta" frameworks like [RedwoodJS](https://redwoodjs.com/) and [Blitz](https://blitzjs.com/). The user interface itself is crafted with the ever popular [Tailwind CSS](https://tailwindcss.com/), a powerful "utility-first" library that lets you compose your styles into higher-level abstractions that you apply across your user interface to give a consistent look and feel. If you would like to implement Dark Mode on a Next.js site using TailwindCSS, let me show you how. It involves three key pieces: * Tailwind's `dark` class * The `Script` tag that we got in Next.js 11 * Understanding, like really understanding, React's `useEffect` hook ## Activating Tailwind's Dark Mode Support Tailwind CSS offers [two ways to set Dark Mode](https://tailwindcss.com/docs/dark-mode). If you are content to default to system settings, then all you need to do is confirm your `tailwind.config.js` file has the `media` setting, which uses the `prefers-color-scheme` [CSS media feature](https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-color-scheme): ~~~js // tailwind.config.js module.exports = { darkMode: 'media', } ~~~ But since we want more control to let Vidya users decide which look they prefer, we need the `class` setting instead: ~~~js // tailwind.config.js module.exports = { darkMode: 'class', } ~~~ Now you need to handle variants like [the TVA in Loki](https://www.gamesradar.com/loki-lady-loki-kid-loki-king-loki-disney-plus/). [Variants in Tailwind](https://tailwindcss.com/docs/configuring-variants) define the ways in which you want to apply different styles. For example, if we want to set a red background on a link hover, we apply the `hover` variant on the `bg` plugin: ``. As an aside, the CSS equivalent would be this for our shade of red: ~~~css a:hover { background-color: #9C4D61; } ~~~ We will do similar to apply `dark` variants of our branding scheme throughout our interface. For example, here is a simplified version of our `contact-us` class composing numerous Tailwind utilities in Next.js's `globals.css` file: ~~~css .contact-us { @apply dark:text-red dark:hover:text-blue bg-red dark:bg-red-light hover:bg-blue-dark dark:hover:bg-blue-light; } ~~~ Note that you always put `dark` first when you have multiple variants like `dark:hover:bg-blue-light`. This is where you will spend most of your time. Mostly because you want to put together a Dark Mode color palette that is usable and accessible and consistent with your branding and because you want to be thorough in applying it throughout the site. Just remember to [extract components](https://tailwindcss.com/docs/extracting-components) as we did above to keep things maintainable, consistent, and organized. Because we are relying on the Tailwind `class` setting for Dark Mode, we need to figure out a way to hook the `dark` class onto the root element of each page like this: ~~~html ... ~~~ And we need to be able to do it on demand. This is where our code comes into play. ## The Script Tag If you've built a website with a lot of client side business functionality, GDPR or other consent management, Google Analytics, social media, or ads, you already know that managing JavaScript execution has always been awkward. Where do you put this script on the page relative to that one? Do you put this script at the top of the `head` element or at the bottom of the `body` element? It's actually easier figuring out where to seat everyone at your wedding. In v11.0.0, Next.js introduced the `Script` [tag](https://nextjs.org/docs/basic-features/script), and it makes all this a lot better. You can put the `Script` tag anywhere, and you apply one of three strategies to let Next.js know when it should execute. Before we specify which strategy should apply here, keep in mind our goal: to assess the user's Dark Mode preference and apply it immediately. For this script to work, it must execute *before* the browser paints the page, so it has to block interactivity. This contradicts everything you've ever read about script optimization. Conventional guidance dictates scripts should run in an asynchronous, parallel fashion in order to maximize [Web Vitals](https://web.dev/vitals/) and get the user up and running as soon as possible. That general guidance is accurate, but we need to make an exception for this particular script. Still, it must execute very quickly, or we will lose customers. Our strategy for implementing Dark Mode will factor in potential user preferences specific to the Vidya website set in `localStorage`, a [key-value store available in modern browsers](https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage), and/or system settings that the browser will inform us with `prefers-color-scheme`. The algorithm goes like this: *If the user previously visited the Vidya website and indicated a preference for Dark Mode OR if there is no preference established and system settings are set for Dark Mode, then activate Dark Mode by attaching the dark class attribute to the root. Otherwise, apply Light Mode by removing any dark class attribute.* Here is the `darkMode.js` script that does exactly that: ~~~js if (localStorage.getItem('vidyaDarkMode') === 'true' || (!('vidyaDarkMode' in localStorage) && window.matchMedia('(prefers-color-scheme: dark)').matches)) { document.documentElement.classList.add('dark') } else { document.documentElement.classList.remove('dark') } ~~~ That's a straightforward conditional, which might even [short-circuit](https://medium.com/@amaliesmidth/javascript-short-circuit-conditionals-6606bdeaa30d), and DOM manipulation. That should be fast. Phew! And here is how we execute it before browser paint with Next.js's `Script` tag inside our `_app.tsx`: ~~~js import Script from "next/script"; // ...