Senior Serverless Spark Migration Engineer

<h2>Senior Platform Engineer, Cloud Infrastructure</h2><p><strong>Type:</strong> Remote (Brazil, Mexico)<br><strong>Coverage:</strong> Pacific Hours (8:00 AM – 5:00 PM PST)</p><hr><h2>About Virtasant</h2><p>Virtasant is a global cloud and technology services company helping organizations modernize, optimize, and build at scale. We work with enterprise customers on complex cloud, data, infrastructure, and AI initiatives, bringing together deep technical expertise and hands-on delivery.</p><h3>About the Role</h3><p>We’re looking for a <strong>Senior Serverless Spark Migration Engineer</strong> to help modernize a large-scale enterprise data platform.</p><p>You’ll lead the migration of production <strong>Apache Spark workloads from on-premise Hadoop/Spark environments to cloud-native and serverless architectures across AWS and GCP</strong>. This is a hands-on engineering role spanning workload assessment, architecture, application refactoring, migration execution, performance optimization, automation, and production readiness.</p><p>The goal is not simply to lift and shift existing workloads. You’ll determine the right target architecture for each workload and establish repeatable patterns that can eventually support migration at significant enterprise scale.</p><h3>What You’ll Do</h3><ul><li><p>Lead migrations of enterprise Spark workloads from on-premise environments to <strong>AWS and GCP</strong>.</p></li><li><p>Assess Spark applications, clusters, configurations, dependencies, data flows, and resource utilization.</p></li><li><p>Determine the right migration approach across <strong>rehost, replatform, refactor, modernize, or retire</strong>.</p></li><li><p>Modernize traditional cluster-based workloads for <strong>serverless Spark</strong> where appropriate.</p></li><li><p>Design and implement architectures using technologies such as <strong>AWS EMR Serverless, S3, Glue, Lake Formation, GCP Dataproc Serverless, GCS, and BigQuery</strong>.</p></li><li><p>Refactor legacy <strong>PySpark/Scala/Spark SQL</strong> applications for cloud portability, scalability, and reliability.</p></li><li><p>Migrate workloads from environments using <strong>Hadoop, HDFS, YARN, Hive, and on-prem Spark clusters</strong>.</p></li><li><p>Troubleshoot and optimize Spark workloads across partitioning, shuffle behavior, joins, data skew, execution plans, executor configuration, serialization, and SQL execution.</p></li><li><p>Benchmark performance and optimize serverless workloads for <strong>performance, reliability, and cloud cost</strong>.</p></li><li><p>Build reusable migration tooling, automation, templates, and frameworks.</p></li><li><p>Implement CI/CD and Infrastructure as Code using tools such as <strong>Terraform</strong>.</p></li><li><p>Define testing, validation, cutover, rollback, observability, and production-readiness patterns.</p></li><li><p>Partner with Data Engineering, ML, Cloud Architecture, Platform Engineering, DevOps/SRE, Security, Governance, and FinOps teams.</p></li></ul><h3>What We’re Looking For</h3><ul><li><p><strong>8+ years</strong> of experience across data engineering, distributed systems, cloud engineering, or platform engineering.</p></li><li><p><strong>5+ years of hands-on Apache Spark experience</strong> in enterprise environments.</p></li><li><p>Strong <strong>PySpark and/or Scala</strong> development experience.</p></li><li><p>Proven experience migrating <strong>large-scale Spark workloads between infrastructure platforms</strong>.</p></li><li><p>Hands-on experience with <strong>both AWS and GCP</strong>.</p></li><li><p>Experience with <strong>on-premise Hadoop/Spark ecosystems</strong>, including technologies such as HDFS, YARN, and Hive.</p></li><li><p>Deep understanding of <strong>Spark internals and distributed processing</strong>.</p></li><li><p>Strong SQL and data engineering fundamentals.</p></li><li><p>Experience with cloud data lakes and object storage.</p></li><li><p>Strong production troubleshooting and performance-tuning experience.</p></li><li><p>Experience with <strong>CI/CD, Git, and Infrastructure as Code</strong>.</p></li><li><p>Ability to own migration work end-to-end, from discovery and architecture through production cutover and optimization.</p></li></ul><h3>Nice to Have</h3><p>Experience with <strong>EMR/EMR Serverless, Dataproc/Dataproc Serverless, Glue, Lake Formation, BigQuery, Delta Lake, Iceberg, Kafka, Airflow, Terraform, Docker, or Kubernetes</strong> is valuable.</p><h3>What Success Looks Like</h3><p>You can take ownership of the complete migration lifecycle:</p><p><strong>Discover → Assess → Design → Refactor → Migrate → Validate → Optimize → Operate</strong></p><p>You understand both the legacy Hadoop/Spark world and modern cloud-native data platforms, and can make pragmatic architecture decisions based on workload characteristics rather than simply reproducing an existing environment in the cloud.</p>

Back to blog

Other Jobs To Apply

Video Editor

[Remote] Motion Designer / Production Artist

UI UX Designer

Senior Digital Graphic Designer Remote 12-14 Week Contract Job at Icreatives in Washington

IT Change, Problem & CMDB Specialist

[Remote] Project Manager (Information Technology)

(Remote) Project Manager - Professional Services

Category Ops Excellence: Materials & Lean Six Sigma

Director, Agile Program & Roadmap Governance

Senior Product Manager - Remote Video Monitoring

Remote Client Portfolio Manager

[Remote] Service Delivery Management Consultant 3- Support

Ecommerce Specialist; Remote

Supplier Relationship Manager - Remote

MRO & Procurement Manager

Remote Management Consultant (Strategy & Operations)

Remote COO AI Operations for Health Equity

Vice President of Operations - EMS Division (1153947)

Remote Scrum Master: Agile Coach for Modernization Delivery

Product Owner, Technical - AI Neobank

Application Product Manager

Technical Recruiter-Trainee (Volunteering Role -Unpaid Opportunity)

Managing Director - Leadership & Assessment

Senior Healthcare Recruiter - Remote

HR Consultant - Remote (DC, Northern Virginia or Maryland)

Rehab Staffing Coordinator at Fairview Health Services: 100% Remote (Shoreview, MN)

Remote Customer Service - Benefits Specialist (50k-90k per year)

Lead Compensation Analyst

Manager, Performance and Test Engineering at BambooHR: 100% Remote (UT)

HRIS Analyst, People Technology – Remote

Marketing & Design Sourcer

[Remote] Learning Development, Operations - Remote

VP of Talent & People Operations

Human Resource Specialist - Freelance AI Trainer Project

People Operations Partner

[Remote] Experienced Operations Recruiter - Remote

Forensic Accountant, Manager/Partner

Senior Analyst, Revenue Recovery

Senior Technical SEO Manager

Remote Entry-Level Bookkeeper (QuickBooks Train...

Head of Finance: Strategic Treasury for Crypto Growth

Remote Quantitative Analyst; Finance

Senior Project Manager EPM (m/f/d) Project-based engagement

Financial Advisor (Remote | )

AU Mortgage Senior End-to-End Credit Analyst (WFH) | ZR_1327_JOB

Salesforce Business Administrator - remote

Senior Tableau Developer (REMOTE)

Digital Solutions Architect- GeoSCADA (Remote on W2)

Remote Full-Stack Engineer (Web3-DeFi)