Software Engineer GEN AI, Assistant Vice President

Citi is the world's most global bank, serving clients in more than 160 countries. Citi is investing heavily in scaled, enterprise-grade generative and agentic AI that operates securely and reliably across a highly regulated financial infrastructure. You'll join the USCC Architecture and AI Engineering team, which owns an enterprise multi-agent AI platform that autonomously supports the software delivery lifecycle, from product analysis through to reviewed, deployable code. 

As a Generative AI Engineer (AVP), you will build and scale production components of this platform. This is a hands-on individual contributor role for an engineer with a strong foundation in computer science, software engineering, and applied AI, ready to ship real production systems rather than one-off prototypes. 

Responsibilities 

• Build and maintain production AI agents covering product analysis, requirements capture, UI prototyping, backlog generation, and automated code implementation with pull request creation. 

• Implement multi-agent workflows using the Google Agent Development Kit (ADK), including sub-agent orchestration, human-in-the-loop approval gates, and agent-to-agent (A2A) messaging. 

• Integrate Gemini and Anthropic Claude into agent pipelines via enterprise gateways, applying advanced prompt engineering and structured tool/function calling. 

• Connect agents to enterprise systems, including Engineering & AI Product Documentation, JIRA, and GitHub, using MCP-style tool interfaces. 

• Build RAG pipelines using vector search, embeddings, and clustering to ground agent responses in accurate enterprise context. 

• Build evaluation harnesses (LLM-as-judge, parallel jury methods) to measure faithfulness, relevancy, hallucination rate, and task completion. 

• Write advanced, async Python and build well-documented FastAPI services, backed by persistent PostgreSQL session state. 

• Apply data structures, algorithms, and complexity analysis to keep orchestration and retrieval logic efficient at scale. 

• Instrument services for observability, maintain CI/CD and containerized deployments, apply secure coding and guardrail practices, and write tests (unit, integration, end-to-end) as part of every change. 

Core Platform & Application Development 

• Full-Stack Engineering & Microservices: Lead the design, implementation, and lifecycle of custom AI platform components. Build production-grade, highly available microservices and full-stack applications using Python, Java (Spring Boot), and modern frontend frameworks (React) for internal LLM/SLM fine-tuning control planes, inference gateways, and experiment tracking dashboards.

• Agentic Frameworks & Patterns: Code, evaluate, and optimize multi-tenant intelligent agents using modular orchestration patterns such as ReAct (Reasoning + Acting) and ReWOO (Reasoning WithOut Observation), leveraging frameworks like Google's Agent Development Kit (ADK), LangGraph, and LangChain. 

• Strict Decoupled Integration Layers: Develop and enforce clean, decoupled enterprise integration architectures.

Software Engineering Fundamentals & Distributed Systems 

• System Architecture & Design Patterns: Apply clean architecture, domain-driven design (DDD), and SOLID principles across Python and Java codebases. Implement robust design patterns (Strategy, Factory, Adapter, Circuit Breaker, Saga) for modularity and fault isolation. 

• Concurrency & High-Throughput Processing: Write robust concurrent and asynchronous code (e.g., Python asyncio, multithreading, Java CompletableFuture/virtual threads, connection pooling) to handle streaming tokens, parallel agent tool execution, and high-concurrency LLM inference loops. 

• Reliability, Fault Tolerance & Resiliency: Design systems for high availability, fault tolerance, and graceful degradation, including rate limiting, token bucket algorithms, retry policies with exponential backoff, and distributed caching strategies (e.g., Redis). 

• Enterprise API Design: Design clean, versioned, idempotent RESTful APIs, gRPC endpoints, and WebSockets (for real-time streaming LLM outputs) following OpenAPI / Swagger specifications. 

  Advanced Data Retrieval & Logic Engineering 

• Next-Generation RAG Architectures: Build and optimize end-to-end data ingestion and retrieval pipelines for advanced RAG paradigms, including Knowledge Graph RAG (GraphRAG), LightRAG, and hierarchical summary trees (RAPTOR). 

• Vector & Graph Integrations: Develop seamless, performant integrations with graph databases (Neo4j) and vector databases (pgvector, Pinecone, Milvus) for complex semantic, hybrid, and relational data retrieval. 

• Prompt & Intent Engineering: Design deterministic LLM system instructions, classification logic, and semantic routing trees to prevent intent collisions and hallucinations in complex multi-step workflows. 

4 Technical Skills & Core Competencies 

A. Software Engineering Fundamentals (Core Pillars)

• Data Structures & Algorithms: Strong command of core data structures (trees, graphs, hash maps, queues, heaps) and algorithmic complexity (Big-O analysis for time and space).

• Object-Oriented & Functional Programming: Good understanding of SOLID principles, clean code, design patterns, separation of concerns, and dependency injection.

• Distributed Systems Concepts: Understanding of CAP theorem, event-driven architectures, asynchronous messaging (Kafka / RabbitMQ), idempotency, distributed transactions, and database isolation levels.

• Testing & Quality Assurance: Comprehensive testing discipline across all layers: Unit testing (pytest, unittest, JUnit, Mockito
• Integration testing, contract testing, and mocking

• Test-Driven Development (TDD) principles

• LLM evaluation and regression testing frameworks

• CI/CD & DevOps Practices: Proficiency with Git workflows, automated CI/CD pipelines, containerization (Docker), container orchestration (Kubernetes / OpenShift), and infrastructure as code.

• Observability & Telemetry: Hands-on experience instrumenting distributed tracing, structured logging, and metrics using OpenTelemetry, Prometheus, Grafana, and ELK/Loki for tracking LLM token latency and tool calls.

• Security & Governance: Understanding of OWASP Top 10, OWASP Top 10 for LLMs, secure API authentication/authorization (OAuth2, OIDC, mTLS, RBAC), secrets management, and data sanitization/PII redaction.

B. AI/ML, Agentic & GenAI Frameworks

• Language & Orchestration: Advanced Python proficiency, prompt engineering, Google ADK, LangChain, LangGraph, and custom state machines.

• Protocol & Standards: In-depth implementation knowledge of Model Context Protocol (MCP) clients and servers.

• RAG & Indexing: GraphRAG, LightRAG, RAPTOR, hybrid search (BM25 + Dense embeddings), re-ranking models (Cross-Encoders), and chunking strategies.

C. Backend, Frontend & Microservices

• Backend: Python (FastAPI, Flask, Django) and Java (Spring Boot, Spring Cloud).

• Frontend: Modern React (hooks, state management, component architecture), TypeScript/JavaScript, and responsive UI design.

D. Data & Storage 

• Relational Databases: Oracle, PostgreSQL (schema design, indexing, query optimization, connection management).

• Vector & Graph Stores: Neo4j (Cypher queries), pgvector, Pinecone, Milvus.

• Caching & Message Brokers: Redis, Apache Kafka.

5 Use Case Development & Forward Deployment (FDE)

• SME Collaboration: Act as a Forward Deployed Engineer (FDE), collaborating directly with Subject Matter Experts (SMEs) to translate business and domain logic into automated, production-ready, agent-driven software.

• End-to-End System Integration: Partner with UI engineers and core API platform teams to connect backend agentic logic flawlessly with user-facing React applications and existing enterprise service layers. 

• Problem Solving & Communication: Strong analytical skills with the ability to clearly communicate complex technical concepts

Required Qualifications & Skills 

• Strong computer science fundamentals: data structures, algorithms and complexity, operating systems, networking, and relational database design/optimization. 

• Advanced Python (async, typing, dependency management) and solid software engineering practices: clean code, automated testing, version control, and CI/CD. 

• Hands-on experience with LLM application development (prompt engineering, tool/function calling, structured outputs) and multi-agent/agentic frameworks. 

• Practical RAG pipeline experience using vector databases and embedding-based retrieval, plus familiarity with structured LLM/agent output evaluation methods. 

• Experience integrating AI systems with external tools/documentation via structured interfaces (e.g., MCP), and working knowledge of Docker and Kubernetes/OpenShift. 

Beneficial Skills & Qualifications 

• Direct experience with Google ADK or comparable orchestration frameworks; observability tooling (Langfuse, Prometheus, distributed tracing). 

• Exposure to Go, and front-end development with React/TypeScript (including SSE streaming). 

• Experience with ChromaDB/pgvector, HDBSCAN clustering, HashiCorp Vault, and SSO/JWT authentication patterns. 

• Experience with GitHub, JIRA, and documentation-search integrations for backlog and pull request workflows. 

Required Experience & Education 

• • Bachelor’s or Master’s degree in Computer Science, Software Engineering, or a related technical discipline. 
  • 4-7 years of professional software engineering experience building scalable backend or full-stack distributed systems. 
  • 3+ years of experience in AI/ML, with at least 2+ years specifically in Generative AI (LLMs, RAG, Agentic workflows, MCP). 

------------------------------------------------------

Job Family Group:

------------------------------------------------------

Job Family:

------------------------------------------------------

Time Type:

------------------------------------------------------

Primary Location:

------------------------------------------------------

Primary Location Full Time Salary Range:

In addition to salary, Citi’s offerings may also include, for eligible employees, discretionary and formulaic incentive and retention awards. Citi offers competitive employee benefits, including: medical, dental & vision coverage; 401(k); life, accident, and disability insurance; and wellness programs. Citi also offers paid time off packages, including planned time off (vacation), unplanned time off (sick leave), and paid holidays. For additional information regarding Citi employee benefits, please visit citibenefits.com. Available offerings may vary by jurisdiction, job level, and date of hire.

------------------------------------------------------

Most Relevant Skills

------------------------------------------------------

Other Relevant Skills

------------------------------------------------------

Anticipated Posting Close Date:

------------------------------------------------------

Automated Processing and AI

We use automated processing, including artificial intelligence, for our legitimate business interests (or our reasonable and appropriate business purposes) to identify and align the candidate's skills and abilities with a specific job opening. Additionally, if you so choose, or consent, we can match your skills and abilities to other suitable roles at Citi.

Importantly, all our hiring processes and decisions, including determining your suitability for a role, are conducted, checked, and decided by individuals. Our automated processing and AI do not involve relying on automatic or autonomous decision-making. Please refer to any Jurisdictional Considerations, with specific provisions for your country (where relevant) for further details.

Illinois residents – AI Notice and Right

------------------------------------------------------

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.

If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.

View Citi’s EEO Policy Statement and the Know Your Rights poster.