Data Engineer – Azure Databricks and Cloud Data Solutions (Safety)

Job Title: Data Engineer – Azure Databricks and Cloud Data Solutions (Safety)

Why this role

Southern Company’s Operations Data Analytics team supports the data foundations and integration capabilities needed to deliver reliable analytics and reporting across multiple organizations. This Data Engineer role will focus on Safety data and be responsible for creating data pipelines, curating source schemas, and supporting data product development in the Azure Databricks lakehouse platform. 

The role is important in creating secure, reusable, and high-performing data services that align with business needs, Technology Organization strategy, and enterprise standards. You will work directly with business and technology stakeholders to turn data requirements into practical solutions while improving performance, resource utilization, capacity planning, and application and data security. This role will support multiple business units and work in multiple lakehouse environments with data provisioned through Unity catalog, to create a unified foundation of safety and human performance data.

About the role

As part of this team, you will design, develop, implement, integrate, and support cloud and data solutions, with hands-on responsibility for data pipelines, data models, ETL processes, lakehouse capabilities, and data storage layers.

You will partner with stakeholders, developers, and infrastructure teams to align solutions across technology platforms. The role also serves as a subject matter expert for data architecture and Databricks, supports operational issue resolution, and helps ensure that project documentation and change management requirements are met.

What you will do

• Design and develop robust data pipelines to ingest, store, and process data from multiple sources for analytics and reporting.

• Create and maintain logical and physical data models that organize data for efficient querying, analysis, and visualization.

• Build and maintain Extract, Transform, Load (ETL) processes that move data between systems while supporting data quality and consistency.

• Implement and support lakehouse solutions using medallion architecture, including data engineering pipelines and data storage layers.

• Optimize data lake storage, retrieval, performance, and resource utilization for analytical workloads.

• Develop dynamic, metadata-driven frameworks and scalable data ingestion patterns.

• Support streaming datasets and Databricks Delta Live Tables.

• Apply lakehouse performance-tuning techniques such as partitioning, aggregations, and materialized views.

• Architect, develop, implement, integrate, and support cloud technology solutions that meet business partner expectations for quality, cost, and timeliness.

• Work closely with developers and infrastructure teams to optimize application code, create reusable services, plan capacity, and protect application and data security.

• Become the team’s subject matter expert for data architecture and Databricks.

• Communicate effectively and promptly with customers, team members, and management, and build strong relationships with peers and business partners.

• Develop required project documentation, follow Change Management policies and procedures, and respond quickly and effectively to operational problems.

• Build a strong understanding of the company’s business challenges and goals and exhibit Our Values.

What success looks like

• Reliable, efficient data pipelines that support analytics and reporting needs.

• Reusable and well-aligned solutions across on-premises and Azure technology platforms.

• Consistent, high-quality data movement through maintainable ETL and lakehouse processes.

• Optimized application code, cloud resources, performance, and capacity utilization.

• Secure data and application solutions supported by complete project documentation and disciplined change management.

• Strong working relationships and timely communication with business partners, technology teams, and management.

• Fast, effective resolution of operational problems and growing team expertise in data architecture and Databricks.

What you bring (required)

Education: Bachelor’s degree required, preferably in Computer Science, Management Information Systems, Engineering, or another technology-related field.

Databricks: Experience with Databricks, including notebooks, clusters, and jobs.

Cloud platforms: Experience with a major cloud platform, preferably Microsoft Azure, and services including Cosmos DB, Storage Application Services, and Azure Data Factory.

Architecture: Experience architecting data-oriented solutions using cloud data architecture platforms and patterns, preferably Azure.

Lakehouse engineering: Experience implementing lakehouses with medallion architecture, data engineering pipelines, and data storage layers.

Data ingestion: Experience with dynamic metadata-driven frameworks and large-scale data ingestion patterns.

Data modeling: Experience developing complex logical and physical data models.

Performance and cost: Experience optimizing cloud PaaS resources to follow best practices and reduce cost, plus lakehouse performance tuning using partitioning, aggregations, materialized views, or similar techniques.

Delivery practices: Experience with Agile, DevOps, and CI/CD methodologies.

Data technologies: Familiarity with Apache Spark, Delta Lake, and MLflow, plus knowledge of structured and unstructured data across databases, data warehouses, and data lakes.

Software development: Knowledge of software development technologies, methodologies, and associated tools.

Problem solving: Excellent root cause analysis and problem-solving skills.

What sets you apart (preferred)

• Hands-on experience with Microsoft Azure data services and cloud PaaS architecture and resources.

• Familiarity with streaming datasets and Databricks Delta Live Tables.

• Strong technical writing skills.

What we value

Customer focus: You deliver cloud and data solutions that meet or exceed business partner expectations for quality, cost, and timeliness.

Ownership and responsiveness: You take responsibility for documentation, change management, and rapid resolution of operational problems.

Collaboration: You communicate effectively and build productive relationships with customers, peers, management, developers, infrastructure teams, and business partners.

Continuous learning: You become a subject matter expert, stay close to the business, and seek to understand the company’s challenges and goals.

Enterprise mindset: You design secure, reusable, aligned solutions and exhibit Our Values.