Costco IT is responsible for the technical future of Costco Wholesale, the third largest retailer in the world with wholesale operations in fourteen countries. Despite our size and explosive international expansion, we continue to provide a family, employee centric atmosphere in which our employees thrive and succeed.
This is an environment unlike anything in the high-tech world and the secret of Costco’s success is its culture. The value Costco puts on its employees is well documented in articles from a variety of publishers including Bloomberg and Forbes. Our employees and our members come FIRST. Costco is well known for its generosity and community service and has won many awards for its philanthropy. The company joins with its employees to take an active role in volunteering by sponsoring many opportunities to help others.
Come join the Costco Wholesale IT family. Costco IT is a dynamic, fast-paced environment, working through exciting transformation efforts. We are building the next generation retail environment where you will be surrounded by dedicated and highly professional employees.
The Principal Engineer is the lead Architect and hands-on builder of our unified AI Platform as a Service. This role is responsible for transforming raw foundation model capabilities into a scalable, multi-tenant reasoning stack that empowers the entire enterprise to build, deploy, and manage semantic discovery, conversational intelligence, and autonomous agents. This role will balance 40% hands-on systems development with 60% platform strategy, personally coding the core orchestration engines, standardized capability servers, and universal trust guardrails. The mission is to provide a central 'AI Operating System' for the company, ensuring that specialized agents across different business units can communicate via inter-agent protocols, access grounded knowledge layers securely, and execute autonomous tasks within a governed, high-performance agent runtime environment.
This position will partner with lead Engineers and Developers to establish reusable patterns and toolsets in order to reduce inconsistencies and increase supportability. This Principal Engineer is responsible for the strategic design, installation, configuration, and operations of API management, gateways (Apigee, DataPower), enterprise service buses (ESB), event-driven Pub/Sub frameworks (Apache Kafka), message queuing (IBM MQ), and registries (WSRR).
These components are core to Costco's API-First integration strategy and our vision to become an industry-leading, data-driven, AI-enabled enterprise. This position will play a crucial role in establishing the integration architecture required to support our Enterprise Wide AI Operating Model, including connecting legacy data to our Google Cloud-powered Unified Data Platform (UDP). Security configuration and management are hardened into all of these components.
This role collaborates across teams to define and document evolving integration infrastructure requirements, and provides enterprise leadership for the ongoing development and maturity of API, data, and intelligent-systems integration governance. A fluency with AI and automation capabilities is expected: this leader actively evaluates, adopts, and champions AI-assisted tooling and practices that improve team velocity, code quality, and operational insight.
If you want to be a part of one of the worldwide BEST companies “to work for”, simply apply and let your career be reimagined.
ROLE
Acts as a team lead and mentors others, bridging the gap between legacy system specialists and modern full-stack developers.
Fosters a culture of continuous learning and AI fluency, encouraging the team to actively experiment with and adopt AI-assisted developer tools, intelligent automation, and emerging integration patterns.
Models responsible adoption of AI capabilities, helping teammates distinguish high-value use cases from hype and navigate the governance and security considerations that come from them.
Follows enterprise standards for security and best practices.
Understands and adheres to Costco’s project methodology and framework.
Seeks opportunities to learn, automate, document, share, educate, and improve processes where appropriate.
Collaborates with infrastructure and/or provisioning groups to stand up the platform.
Participates in the creation of documentation and artifacts used to describe the mechanisms used for deployment, monitoring, and maintenance.
Conducts research and makes recommendations on standards, products, and services.
Ensures application and infrastructure architectural solutions are stable, secure, and compliant with company standards and practices as well as regulatory requirements.
AI and Intelligent Systems Integration
Architects integration pipelines to securely feed enterprise data into AI applications, supporting key strategic initiatives like our Fraud Detection AI and Contact Center Modernization utilizing Agentic AI.
Partners with the data science team to establish API endpoints for machine learning models hosted on Google Vertex AI (Endpoints, Model Garden, and Predictions).
Designs event-driven and API based data flows that support real-time and batch ingestion into AI/ML training and inference pipelines, enabling accurate, fresh and well governed model inputs.
Evaluates and adopts AI assisted developer productivity tools (e.g. Github Copilot, Gemini Code Assist) and quantifies their impact on delivery quality and throughput.
Stays current on the evolving landscape of AI integration patterns, proactively sharing knowledge and raising the AI fluency of the broader engineering organization.
Technical Leadership and Operations
Interfaces with and provides technical leadership to others in the IT division and business to address ongoing business needs.
Documents the application infrastructure and teaches/shares with others as necessary.
Serves as senior escalation point for complex issues and ensures timely recovery from outages, performs root cause analysis, and implements preventative measures.
Monitors integration infrastructure health and incorporates AI-assisted observability and anomaly detection capabilities where appropriate.
Solution Planning:
Works with vendors, business, and application development teams, and other technical teams to meet the needs of the project.
Works with architects and Solution Analysts to develop, implement, and support new capabilities and designs.
Assesses all project information to understand the scope, current IT and business environment, objectives, and priorities.
Defines and documents the proposed, high-level solution, transitioning legacy data interactions to REST APIs and Event-Driven architectures to support AI-readiness and our Infrastructure 2.0 mandate.
Defines and documents the proposed, high-level solution, including the use of Commercial Off The Shelf (COTS) packages.
Analyzes technical risks and advises on risk mitigation strategies, including risks specific to AI data pipeline integrity, model input quality, and governance.
Solution Delivery:
Assists/provides project managers with work breakdown of tasks and effort estimates.
Takes responsibility for the technical content (architecture and design), integrity, and quality of the solution.
Takes accountability for the delivery of the solution as specified (by requirements and architecture).
Creates architectural models that mandate the decoupling of legacy DB2 systems to streamline data supply for AI and machine learning ingestion pipelines.
Establishes high-level models that guide solution architecture design, sub-architecture, or deployment and reviews with all interested parties.
Defines and implements plans to address the service's integration points with other services.
REQUIRED
7+ years’ experience installing, configuring, and maintaining DataPower, Apigee, or other similar technologies.
7+ years’ experience deploying services or APIs.
7+ years’ working with SAML, XACML, OAUTH security standards.
7+ years’ designing service registries and repositories.
7+ years’ scripting to automate operational tasks.
Ability to quickly understand current environments (including legacy systems) and use that information to build stable, responsive, and secure solutions.
Understanding of Identity Access Management patterns, concepts, and best practices.
Experience building solutions in a distributed hybrid multi-cloud architecture and ability to understand interactions from end to end
Solid understanding of ESB patterns, Data Integration Services patterns, and middleware tools (WAS, etc.) patterns, concepts, and best practices.
Proven ability to lead various teams within the development lifecycle.
Process improvement skills and demonstrated ability to effectively troubleshoot and provide solutions.
Solid negotiation, delegation, and team-building skills.
Demonstrated AI fluency: comfort learning and working alongside AI tools, evaluating their outputs critically, and guiding a team through responsible adoption.
Experience integrating with cloud-native AI platforms such as Google Vertex AI, Google Document AI, or Azure AI Search.
Hands-on experience with AI-assisted developer tools like GitHub Copilot, Gemini Code Assist, or Microsoft Copilot in Power Automate
Hands-on experience with modern AI integration patterns, specifically focusing on data provenance for RAG pipelines and managing safe, bounded data traversals for intelligent workflows
Proficient in Active Directory, Access Management and Provisioning, ESB integration patterns and implementations, SAML, WMQ (WebSphere MQ / IBM MQ), WMB (WebSphere Message Broker / IIB), WSRR (WebSphere Service Registry and Repository), Informatica, and WAS (WebSphere Application Server).
Experience with event-driven data streaming platforms such as Apache Kafka.
Experience with SOAP, REST, XML, JSON, XSLT, Javascript, Java.
Familiarity with application and service integration patterns.
Experience with legacy and partner integration platforms (ie, Axway, EDI)
Azure certifications.
Experience with Kubernetes.
Good understanding of CI/CD and Azure DevOps.
Familiarity with Tivoli and other monitoring technologies.
Proficient in Google Workspace applications, including Sheets, Docs, Slides, and Gmail.
Required Documents
California applicants, please review the Costco Applicant Privacy Notice.
Pay Range: $160,000 - $230,000, Bonus and Restricted Stock Unit (RSU) eligible
We offer a comprehensive package of benefits including paid time off, health benefits - medical/dental/vision/hearing aid/pharmacy/behavioral health/employee assistance, health care reimbursement account, dependent care assistance plan, short-term disability and long-term disability insurance, AD&D insurance, life insurance, 401(k), stock purchase plan to eligible employees.
Costco is committed to a diverse and inclusive workplace. Costco is an equal opportunity employer. Qualified applicants will receive consideration for employment without regard of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or any other legally protected status. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to
[email protected]
If hired, you will be required to provide proof of authorization to work in the United States. Applicants and employees for this position will not be sponsored for work authorization, including, but not limited to H1-B visas.
