Data Engineer

Position Details:

Title: Data Engineer

Location - Irving, TX

Long term Engagement

Primary Responsibilities:

• Design and build large scale data processing system (real-time and batch) to address growing AI/ML and Data needs of a Fortune 500 company
• Build a product to process large amount data/events for AI/ML and Data consumption
• Automate test coverage (90+%) for data pipelines. Best practices and frameworks for unit, functional and integration tests.
• Automate CI and deployment processes and best practices for the production data pipelines.
• Build AI/ML model based alert mechanism and anomaly detection system for the product. The goal is have a self-annealing product

Required Skills/Experience

• 7+ years of overall experience in software development with 5 or more years of relevant experience in designing, developing, deploying and operating large data processing data pipelines at scale.
• 3 or more years experience with Apache Spark for Streaming and batch process
• Good knowledge on Apache Kafka
• Strong background in programming (Scala/Java)
• Experience on building reusable data frameworks/modules
• Experience on Airflow scheduler
• Experience with Containers, Kubernetes and scaling elastically
• Strong background in algorithms and data structures
• Strong analytical and problem solving skills
• Strong bent towards engineering solutions which increase productivity of data consumers
• Strong bent toward completely automated code deployment/testing (DevOps, CI/CD)
• Passion for data engineering and for enabling others by making their data easier to access.
• Some experience with working with and operating workflow or orchestration frameworks, including open source tools like Activiti, Spring Boot, Airflow and Luigi or commercial enterprise tools.
• Excellent communication (writing, conversation, presentation) skills, consensus builder
• Demonstrated ability to tackle tough coding challenges independently and work closely with others on a highly productive coding team

Must have Skills: Apache Spark, Apache Kafka, Scala/Java, NoSQL Databases, Elasticsearch & Kibana, Kubernetes, Docker Containers

Nice to have Skills:

• Knowledge of API Development
• Apache Flink experience
• Cloud experience
• DevOps skills
• Any other streaming technologies/tools experience

Dexian is a leading provider of staffing, IT, and workforce solutions with over 12,000 employees and 70 locations worldwide. As one of the largest IT staffing companies and the 2nd largest minority-owned staffing company in the U.S., Dexian was formed in 2023 through the merger of DISYS and Signature Consultants. Combining the best elements of its core companies, Dexian's platform connects talent, technology, and organizations to produce game-changing results that help everyone achieve their ambitions and goals.

Dexian's brands include Dexian DISYS, Dexian Signature Consultants, Dexian Government Solutions, Dexian Talent Development and Dexian IT Solutions. Visit https://dexian.com/ to learn more.

Dexian is an Equal Opportunity Employer that recruits and hires qualified candidates without regard to race, religion, sex, sexual orientation, gender identity, age, national origin, ancestry, citizenship, disability, or veteran status.