Job#: 2047577
Job Description:
Responsibilities:Data Pipeline Development: Design, develop, and maintain robust data pipelines using Databricks to process and transform large volumes of data.
ETL Process Management: Implement ETL (Extract, Transform, Load) processes to integrate data from various sources into Databricks, ensuring data quality and integrity.
Data Integration: Integrate Databricks with other data storage solutions and data lakes, ensuring seamless data flow and accessibility.
Performance Optimization: Optimize data processing and query performance within Databricks to ensure efficient data retrieval and processing.
Data Analysis and Visualization: Utilize Databricks to perform complex data analysis and create visualizations to support data-driven decision-making.
Collaborate with Data Scientists and Analysts: Work closely with data scientists and analysts to understand their requirements and provide the necessary infrastructure and tools within Databricks.
Security and Compliance: Ensure that data processing within Databricks complies with organizational security policies and industry regulations, implementing necessary security measures. This includes setting up encryption, managing network security configurations, and performing regular security audits.
Monitoring and Troubleshooting: Monitor data pipelines and workflows for performance issues or errors, and troubleshoot any problems that arise to maintain smooth operations.
Cluster Management: Manage the creation, configuration, and scaling of Databricks clusters to ensure optimal performance and cost-efficiency. This includes monitoring cluster usage, resource allocation, and ensuring high availability.
User and Access Management: Implement and manage user access controls, ensuring that only authorized personnel have access to Databricks resources. This involves setting up role-based access controls (RBAC), managing permissions, and integrating with identity management systems.
Backup and Disaster Recovery: Develop and implement backup and disaster recovery plans for Databricks environments. Ensure that data and configurations are regularly backed up and that there are clear procedures in place for restoring services in the event of a failure.
Required Qualifications:
Technical Skills
- Experience with Databricks: Hands-on experience with Databricks, including familiarity with its architecture, features, and services.
- Proficiency in Spark: Strong knowledge of Apache Spark, including Spark SQL, Spark Streaming, and Spark MLlib, as Databricks is built on Spark.
- Programming Languages: Proficiency in programming languages commonly used in data engineering such as Python, Scala, SQL, and Java.
- Data Warehousing and ETL: Experience with data warehousing concepts, ETL processes, and tools like Apache Airflow, Talend, or Informatica.
- Database Management: Knowledge of relational and NoSQL databases, data modeling, and query optimization.
- Big Data Technologies: Familiarity with big data technologies and ecosystems, including Hadoop, Hive, and Kafka.
- Data Analysis: Ability to perform complex data analysis and create data visualizations to support business decisions.
- Problem-Solving: Strong analytical and problem-solving skills to troubleshoot and resolve issues in data pipelines and workflows.
- Communication Skills: Excellent verbal and written communication skills to collaborate with data scientists, analysts, and other stakeholders.
- Team Collaboration: Ability to work effectively in a team environment and contribute to cross-functional projects.
- Databricks Certifications: Certifications such as Databricks Certified Associate Developer for Apache Spark or Databricks Certified Professional Data Scientist can demonstrate expertise and enhance job prospects.
- Cloud Certifications: Certifications from cloud providers (e.g., Azure Certified Solutions Architect, Azure Data Engineer) can be advantageous.
Relevant Experience: Prior experience working in data engineering, data analytics, or a related field is often required. This includes experience in building and maintaining data pipelines, ETL processes, and data integration.
EEO Employer
Apex Systems is an equal opportunity employer. We do not discriminate or allow discrimination on the basis of race, color, religion, creed, sex (including pregnancy, childbirth, breastfeeding, or related medical conditions), age, sexual orientation, gender identity, national origin, ancestry, citizenship, genetic information, registered domestic partner status, marital status, disability, status as a crime victim, protected veteran status, political affiliation, union membership, or any other characteristic protected by law. Apex will consider qualified applicants with criminal histories in a manner consistent with the requirements of applicable law. If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation in using our website for a search or application, please contact our Employee Services Department at [email protected] or 844-463-6178.
Apex Systems is a world-class IT services company that serves thousands of clients across the globe. When you join Apex, you become part of a team that values innovation, collaboration, and continuous learning. We offer quality career resources, training, certifications, development opportunities, and a comprehensive benefits package. Our commitment to excellence is reflected in many awards, including ClearlyRated's Best of Staffing® in Talent Satisfaction in the United States and Great Place to Work® in the United Kingdom and Mexico.