Job Description
ASSYST is seeking a Senior Data Engineer with expertise in Azure Synapse Analytics and Microsoft Fabric to build, manage, and optimize scalable data pipelines and solutions. The role focuses on developing high-performance data architectures to support analytics and reporting needs across the Azure ecosystem. The candidate will set up and manage data lakes for both structured and unstructured data, ensuring high availability, security, and scalability.
The ideal candidate will collaborate with cross-functional teams, including analysts and data scientists, to gather data requirements and deliver efficient, scalable data solutions. Strong knowledge of ETL processes, cloud technologies, data governance, and performance tuning is essential. Additionally, the engineer will support real-time data ingestion and multi-cloud integration to enable seamless analytics across platforms.
Job Title: Senior Data Engineer
Location: Laurel, MD 20707(Hybrid - 3 days onsite 2 days remote)
Responsibilities: - Build and optimize complex data pipelines using Azure Data Factory and Synapse Analytics to extract, transform, and load data efficiently.
- Develop and maintain Azure Data Lake solutions to store and manage large volumes of structured and unstructured data.
- Integrate data from multiple sources, such as relational databases, NoSQL databases, APIs, and flat files, into Azure for reporting and analytics.
- Design and implement scalable data architectures for Azure Synapse Analytics and Azure Data Lake environments.
- Develop ETL/ELT processes to ensure timely and accurate data movement and transformation across the Azure ecosystem.
- Monitor and troubleshoot data pipelines to ensure high availability and minimize downtime and inconsistencies.
- Continuously optimize data pipelines and query performance for large datasets and complex transformations within Synapse Analytics.
- Collaborate with data scientists, analysts, and stakeholders to design scalable data solutions that meet business needs.
- Implement security best practices, including access control and data governance policies, within Azure environments.
- Design logical and physical data models to enable high-performance querying in Synapse Analytics.
- Apply data quality checks, validation processes, and error handling to ensure data integrity and consistency.
- Manage data governance frameworks, tracking data lineage, metadata, and ensuring compliance with policies and regulations.
- Implement data partitioning and indexing strategies to improve query performance in data lakes and Synapse Analytics.
- Automate data workflows using Azure tools to enable end-to-end data processing and transformation.
- Maintain detailed documentation for data architectures, pipelines, and models for easy maintenance and transparency.
- Provide mentorship and technical guidance to junior engineers, ensuring adherence to best practices.
- Monitor cloud resource usage and plan for scalable data growth to ensure efficient use of resources.
- Design and manage multi-cloud solutions, integrating data from platforms like AWS and OCI into Azure Synapse.
- Implement hybrid data architectures that synchronize on-premises data centers with Azure cloud environments.
- Utilize tools like Azure Purview to create an enterprise-wide data catalog for efficient data discovery.
- Build machine learning pipelines that integrate data ingestion, feature engineering, model training, and deployment.
- Use Python for creating and scheduling data pipelines and data augmentation techniques to generate synthetic datasets for machine learning.
- Implement real-time data ingestion using Azure Stream Analytics and Event Hubs.
- Design and support a data mesh architecture to enable decentralized data ownership and self-service data management.
Preferred Experience & Qualifications: - Bachelor’s Degree in Information Systems, Computer Science, or a related technical field.
- Minimum 5 years of relevant experience in designing and implementing enterprise-level data solutions.
- Experience with ETL processes, data modeling, and managing large-scale data projects.
- Industry experience in water and wastewater utilities with knowledge of Oracle utility applications is preferred.
- Proven experience working on analytics, business intelligence, and enterprise systems in diverse team environments.
- Strong understanding of data modeling, database design, and data governance frameworks.
- Knowledge of SQL, PL/SQL, and Oracle databases with performance optimization expertise.
- Experience in data lakes and data warehouse design.
- Proficiency in Python for managing data pipelines and automating workflows.
- Familiarity with multi-cloud and hybrid data architectures, integrating data across platforms.
- Knowledge of Medallion architecture, parquet and delta file structures, and working with non-structured data (e.g., XML, JSON).
- Excellent communication and collaboration skills to translate business needs into technical solutions.
- Strong analytical and problem-solving abilities, especially when working with large and complex datasets.
ASSYST is an Equal Opportunity Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, disability, military status, national origin or any other characteristic protected under federal, state, or applicable local law.
Job Tags
Local area, Remote job,