Job Description
Role: Data Engineering Lead - AWS & Databricks
Full time Professional
Remote - Nationwide
About Onix: Onix is a leading cloud consulting company that helps customers get the most out of their technology with innovative, cloud-powered solutions and best-in-class services. We offer cloud solutions across various use cases and industries, from migration to advanced analytics and innovative AI solutions. With our global team of accomplished cloud experts, we are the most trusted cloud partner in the industry.
Summary: Onix seeks an experienced Databricks Engineering lead with a proven track record in production-level Databricks and AWS data engineering implementations to join our delivery team. The ideal candidate will strongly understand Lakehouse Architecture, ETL solutions, data modeling, and data management, focusing significantly on data streaming. They should have a strong background in software development, particularly in building utilities and solutions using pySpark within Databricks.
You will collaborate with cross-functional teams, including client executives, architects, engineers, and business stakeholders, to understand their requirements and translate them into robust cloud solutions. The candidate will have hands-on experience with both Databricks and AWS.
This role required being a self-starter and may include overseeing other engineers' work.
Location: Remote - US, Support EST business hours
Scope/Level of Decision-Making: This exempt position operates under limited decision-making and supervision. Position performs various assigned activities, referring more complex issues to the manager.
Primary Responsibilities:
You will lead the design, development, and implementation of enterprise-level Databricks data engineering solutions, including, but not limited to, developing and optimizing data pipelines, implementing data transformations, and ensuring the efficiency and reliability of data processing workflows.
Provide thought leadership and technical expertise in designing, building, and optimizing end-to-end data pipelines, from data extraction and transformation to loading and visualization.
Collaborate with business stakeholders, data engineers, data scientists, and other teams to understand data requirements, use cases, and business objectives.
Design, implement, and optimize pySpark-based solutions within Databricks
Design, implement, and optimize data pipelines, ETL processes, and data integration workflows using Databricks and related technologies.
Optimize and tune data processing frameworks, data ingestion mechanisms, and data storage strategies for performance and cost efficiency.
Ensure data quality, integrity, and governance standards are met throughout the data engineering lifecycle.
Create and maintain technical documentation, run books, code, and best practices for data and machine learning engineering on the Databricks platform.
Stay up-to-date with the latest advancements in data analytics, Databricks, and related technologies, and propose innovative solutions to enhance our capabilities.
Consult on Professional Services Engagements to help our customers design and implement data warehouse solutions. Lead and develop best practices for the larger data analytics delivery team.
Provide client presentations to review project design, outcomes, and recommendations.
Minimum Qualifications & Skills
6+ years of experience in a customer-facing or consulting role.
7+ years of experience in software development, with a focus on data engineering and analytics.
Experience with Big Data technologies, Analytic Warehousing technologies, data processing technologies, and performance and scalability optimizations.
Experience delivering technical presentations and live demonstrations in an enterprise business environment.
Experience with architecture design, implementing, tuning, schema design, and query optimization of scalable and distributed systems.
Experience developing data warehousing, data lakes, batch/real-time event processing, streaming, data processing (e.g., Extract, transform, and load /Extract, Load, and Transform), data migrations, data visualization tools, and data governance on cloud-native architectures.
Knowledge of cloud computing (e.g., infrastructure, storage, platforms, and data).
Proficiency in pySpark and strong understanding of Apache Spark architecture and concepts.
Proficiency in programming languages such as Python, Scala, Java, Javascript/Typescript, TerraForm, Scala, SQL, or others and experience with related libraries and frameworks.
Experience with cloud platforms like AWS, Azure, or GCP and their services (e.g., S3, BigQuery, etc).
Knowledge of terraform infrastructure as code technology stacks
Exceptional communication and interpersonal skills, with the ability to effectively interact with stakeholders at all levels of the organization.
Preferred qualifications:
Experience as a consultant in the fields of cloud computing, data, information lifecycle management, and Big Data.
Experience in large-scale data and database migrations.
Familiarity with data landscape, pros and cons of leading products.Qualifications & Skills
Certifications:
Databricks Certified Data Engineer Associate
Preferred to have current credentials or ability to obtain within first 60 days of employment
AWS - Solutions Architect Associate or Professional
Preferred to have current credentials or ability to obtain within first 60 days of employment
Education: Bachelor's Degree or equivalent experience required.
Travel Expectation: Up to 25%
It is the policy of Onix to ensure equal employment opportunity in accordance with the Ohio Revised Code 125.111 and all applicable federal regulations and guidelines. Employment discrimination against employees and applicants due to race, color, religion, sex, (including sexual harassment), national origin, disability, age (40 years old or more), military status, or veteran status is illegal. Onix will only employ those who are legally authorized to work in the United States or Canada. This is not a position for which sponsorship will be provided.
Individuals with temporary visas such as E, F-1, H-1, H-2, L, B, J, or TN or who need sponsorship for work authorization now or in the future, are not eligible for hire.
