Lead Data Engineer
Engineering Data Analyst Data Engineer Engineering Data Leadership
Mức lương
Khu vực
Phúc lợi
Lương tháng 13
Thời gian làm việc linh hoạt
Bảo hiểm full lương
Phúc lợi khác
Đánh giá lương hàng năm
Du lịch/Công tác
Cấp laptop
Thưởng chỉ tiêu
Bảo hiểm nâng cao
Tổng quan về công việc và trách nhiệm
As a Data Engineer at GFT, you will be responsible for managing, designing, and enhancing data systems and workflows that drive key business decisions. The role is focused 75% on data engineering, involving the construction and optimization of data pipelines and architectures, and 25% on supporting data science initiatives through collaboration with data science teams for machine learning workflows and advanced analytics. You will leverage technologies like Python, Airflow, Kubernetes, and AWS to deliver high-quality data solutions. Key Activities - Architect, develop, and maintain scalable data infrastructure, including data lakes, pipelines, and metadata repositories, ensuring the timely and accurate delivery of data to stakeholders. - Work closely with data scientists to build and support data models, integrate data sources, and support machine learning workflows and experimentation environments. - Develop and optimize large-scale, batch, and real-time data processing systems to enhance operational efficiency and meet business objectives. - Leverage Python, Apache Airflow, and AWS services to automate data workflows and processes, ensuring efficient scheduling and monitoring. - Utilize AWS services such as S3, Glue, EC2, and Lambda to manage data storage and compute resources, ensuring high performance, scalability, and cost-efficiency. - Implement robust testing and validation procedures to ensure the reliability, accuracy, and security of data processing workflows. - Stay informed of industry best practices and emerging technologies in both data engineering and data science to propose optimizations and innovative solutions.
Kỹ năng và kinh nghiệm tối thiểu
- Bachelor's in Software Engineering or related fields - 12+ years of experience in total - 5+ years as a Data Engineer, hands on Python programming ]- Core Expertise: Proficiency in Python for data processing and scripting (pandas, pyspark), workflow automation (Apache Airflow), and experience with AWS services (Glue, S3, EC2, Lambda). - Containerization & Orchestration: Experience working with Kubernetes and Docker for managing containerized environments in the cloud. - Data Engineering Tools: Hands-on experience with columnar and big data databases (Athena, Redshift, Vertica, Hive/Hadoop), along with version control systems like Git. - Cloud Services: Strong familiarity with AWS services for cloud-based data processing and management. - CI/CD Pipeline: Experience with CI/CD tools such as Jenkins, CircleCI, or AWS CodePipeline for continuous integration and deployment. - Data Engineering Focus (75%): Expertise in building and managing robust data architectures and pipelines for large-scale data operations. - Data Science Support (25%): Ability to support data science workflows, including collaboration on data preparation, feature engineering, and enabling experimentation environments.
Tại sao ứng viên nên làm ở đây
- Competitive Compensation - Benefits package including comprehensive medical, dental, vision and others - Company Culture based on our Core Values - Professional Development Training with Individual Development Plans to map out your career growth - Opportunity to work in a global environment with diverse teams built with colleagues from around the world - Opportunity to work with technology industry leaders in the financial services industry - Opportunity to work for big name clients in capital markets, banking and other industries
Ưu tiên có kỹ năng và kinh nghiệm
- Langchain Experience: Familiarity with Langchain for building data applications involving natural language processing or conversational AI frameworks. - Advanced Data Science Tools: Experience with AWS Sagemaker or Databricks for enabling machine learning environments. - Big Data & Analytics: Familiarity with both RDBMS (MySQL, PostgreSQL) and NoSQL (DynamoDB, Redis) databases. - BI Tools: Experience with enterprise BI tools like Tableau, Looker, or PowerBI. - Messaging & Event Streaming: Familiarity with distributed messaging systems like Kafka or RabbitMQ for event streaming. - Monitoring & Logging: Experience with monitoring and log management tools such as the ELK stack or Datadog. - Data Privacy and Security: Knowledge of best practices for ensuring data privacy and security, particularly in large data infrastructures.
Báo cáo cho
Supervisor
Quá trình phỏng vấn
Technical interview with GFT (live coding included) -> 2 interview with clients (technical)
