Orchestrated a multi-stage serverless workflow in AWS Step Functions, coordinating Lambda tasks and AWS Batch jobs using the built-in service-integration patterns (Request/Response and Run-a-Job .sync) to move products through clustering and post-processing gates.
Implemented environment automation that scales Amazon EKS managed node groups up/down from Lambda during heavy Milvus workloads via update_nodegroup_config, then proceeds only when the node group reports healthy. Boto3
Controlled Milvus availability inside the state machine by calling its REST v2 endpoints: load the collection and poll get_load_state until memory-resident before fanning out downstream tasks. Milvus
Added a “processproducts” branch that generates Azure OpenAI text embeddings for product descriptions and records an execution trail for each run. Microsoft Learn
Drove Batch handoffs with deterministic state transitions by monitoring job status (SUCCEEDED/FAILED/in-progress) and returning structured payloads for the next Step Functions state. AWS Documentation
Wrapped the flow with robust Secrets Manager retrieval, STS role-assumption, and PostgreSQL upserts/updates to keep product status, metadata, and audit logs consistent across retries and reruns