Senior Data Engineer - Python Airflow

Job Description

Are you ready to write your next chapter?

Make your mark at one of the biggest names in payments. With proven technology, we process the largest volume of payments in the world, driving the global economy every day. When you join Worldpay, you join a global community of experts and changemakers, working to reinvent an industry by constantly evolving how we work and making the way millions of people pay easier, every day.

What makes a Worldpayer? It's simple: Think, Act, Win. We stay curious, always asking the right questions to be better every day, finding creative solutions to simplify the complex. We're dynamic, every Worldpayer is empowered to make the right decisions for their customers. And we're determined, always staying open - winning and failing as one.

We're looking for a Senior Data Engineer - Python Airflow to join our ever evolving Cloud Data Platform team to help us unleash the potential of every business.

Are you ready to make your mark? Then you sound like a Worldpayer.

About The Team

Worldpay Cloud Data Platform team provides WorldPay Big Data capabilities including large scale data processing, analytics, fraud platforms, machine learning engineering, stream processing and data insights. Worldpay Cloud Data Platform team help maintain and advance cloud data platforms and services to build data products powering our business services and customers.

What You'll Own

* Develop and implement strategies for data engineering initiatives using Python, AWS, Airflow, and Snowflake technologies

* Monitor trends in the data engineering industry and stay up to date on current technologies

* Collaborate with product team to develop solutions that meet their goals and objectives

* Act as a subject matter expert for Apache Airflow and provide technical guidance to team members

* Install, configure, and maintain Astronomer Airflow environments

* Build complex data engineering pipelines using Python and Airflow

* Will be responsible for designing, developing, and maintaining scalable workflows and orchestration systems using Astronomer

* Create and manage Directed Acyclic Graphs (DAGs) to automate data pipelines and processes

* Leverage AWS Glue, Step Functions, and other services for orchestrating data workflows

* Develop custom operators and plugins for Astronomer to extend its capabilities

* Integrate code with defined CI/CD framework and AWS services required for building secure data pipeline

* Manage user access and permissions, ensuring data security and compliance with company policies

* Implement and monitor security controls, including encryption, authentication, and network security

* Conduct regular security audits and vulnerability assessments

* Manage data ingestion and ETL processes

* Automate routine tasks and processes using scripting and automation tools

* Maintain comprehensive documentation for configurations, procedures, and troubleshooting steps

What You Bring

* 5+ yrs in a pivotal Software/Data Engineering role with deep exposure to modern data stacks, particularly Snowflake, Airflow, DBT, and AWS data services

* Proficiency in cloud platforms such as AWS, Azure, or Google Cloud

* Experience with PySpark/Hadoop and/or AWS Glue ETL and/or Databricks w/ python is preferred. Must have experience on ETL development life cycle, best practices of ETL pipelines, thorough work experience on data warehouse using combination of Python, Snowflake & AWS Services

* Data engineering experience with AWS Services (S3, Lambda, Glue, Lake Formation, EMR), Kafka, Streaming, Databricks is highly preferred

* Experience in Astronomer and/or Airflow

* Understanding data pipelines and modern ways of automating data pipeline using cloud-based testing and clearly document implementations, so others can easily understand the requirements, implementation, and test conditions

* Serving as the Databricks account owner, including security and privacy setup, marketplace plugins and integration with other tools

* Experience Unity Catalog migration, workspaces and audit logs

* Strong experience with Amazon Web Services (AWS) accounts and high-level usage monitoring

* Proficiency in scripting languages such as Python, Bash, or PowerShell

* Experience with CI/CD tools like Jenkins, GitLab CI, CircleCI, or similar

* Familiarity with monitoring and logging tools (e.g., Prometheus, Grafana, ELK stack)

* Excellent problem-solving skills and attention to detail

* Strong communication skills and the ability to work collaboratively in a team environment

* Experience with AWS CLI and Networking

* Experience with architecting and maintaining high availability production systems

* Experience with developing monitoring architecture and implementing monitoring agents, dashboards, escalations, and alerts

* Knowledge of security controls for the public cloud (encryption of data in motion/rest and key management)

* Demonstrated knowledge and hands-on experience with AWS alerting/monitoring tools

* Experience with infrastructure as code (IaC) tools such as Terraform

* AWS certification

* Airflow Certification

* Python Certification

Worldpay Perks - What We'll Bring For You

We know it's bigger than just your career. It's your life, and your world. That's why we offer global benefits and programs to support you at every stage. Here's a taste of what you can expect.

• A competitive salary and benefits

• Time to support charities and give back to your community

• Parental leave policy

• Global recognition platform

• Virgin Pulse access

• Global employee assistance program

What Makes A Worldpayer

At Worldpay, we take our Values seriously, and we live them every day. Think like a customer, Act like an owner, and Win as a team.

• Curious. Humble. Creative. We ask the right questions, listening and learning to get better every day. We simplify the complex and we're always looking to create a bigger impact for our colleagues and customers.

• Empowered. Accountable. Dynamic. We stay agile, using our initiative, taking calculated risks to progress. Never standing still, never settling, we work at pace to achieve our goals. We champion our ideas and stay flexible to make them happen. We know that every action adds up.

• Determined. Inclusive. Open. Unlocking potential means working as one global community. Our work spans borders, and we stay united by our purpose. We collaborate, always encouraging others to perform at their best, welcoming new perspectives.

Does this sound like you? Then you sound like a Worldpayer.

Apply now to write the next chapter in your career. We can't wait to hear from you.

To find out more about working with us, find us on LinkedIn.

Privacy Statement

Worldpay is committed to protecting the privacy and security of all personal information that we process in order to provide services to our clients. For specific information on how Worldpay protects personal information online, please see the Online Privacy Notice.

EEOC Statement

Worldpay is an equal opportunity employer. We evaluate qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, marital status, genetic information, national origin, disability, veteran status, and other protected characteristics. The EEO is the Law poster is available here supplement document available here.

For positions located in the US, the following conditions apply. If you are made a conditional offer of employment, you will be required to undergo a drug test. ADA Disclaimer: In developing this job description care was taken to include all competencies needed to successfully perform in this position. However, for Americans with Disabilities Act (ADA) purposes, the essential functions of the job may or may not have been described for purposes of ADA reasonable accommodation. All reasonable accommodation requests will be reviewed and evaluated on a case-by-case basis.

Sourcing Model

Recruitment at Worldpay works primarily on a direct sourcing model; a relatively small portion of our hiring is through recruitment agencies. Worldpay does not accept resumes from recruitment agencies which are not on the preferred supplier list and is not responsible for any related fees for resumes submitted to job postings, our employees, or any other part of our company.