Contract Data Engineer (Databricks / PySpark) 500pd Inside of IR35 6 weeks (Initially) plus exetension

We're looking for a contract Data Engineer to embed in a small, cross-functional mini squad supporting a range of core projects. You'll take an existing MVP data pipeline and harden it for scale, then help deliver a second MVP focused on ingesting new data sources.

This is a hands-on engineering role on Databricks and PySpark. You'll work closely with the squad day to day, owning engineering quality across the pipelines you touch.

Key responsibilities

Scale the existing MVP pipeline

• Optimise PySpark and SQL code for performance, cost and reliability on Databricks.
• Add monitoring and alerting for pipeline performance, failures and data quality.
• Write clear documentation covering architecture, data flows, runbooks and operational support.
• Improve job scheduling, orchestration and release processes so the pipeline runs predictably at scale.
• Apply engineering best practice: version control, code review, testing and CI/CD through GitLab.

Support the new MVP

• Build ingestion for new data sources, from initial discovery through to production-ready pipelines.
• Design reusable, maintainable patterns that the squad can extend to future sources.
• Work with stakeholders to understand source data, requirements and acceptance criteria.

Squad contribution

• Act as the embedded engineering lead within the mini squad, sharing knowledge and raising standards.
• Contribute to planning, estimation and prioritisation across the core projects the squad supports.

Required skills

• Databricks (DBX): strong commercial experience building and running production pipelines, including jobs, clusters and workflows.
• PySpark: proven ability to write and performance-tune distributed data processing code at scale.
• Python: clean, tested, production-quality code.
• SQL: advanced querying, data modelling and query optimisation.
• Git / GitLab: branching, merge requests, code review and GitLab CI/CD pipelines.
• Experience taking a prototype or MVP pipeline through to a robust, production-grade service.

Desirable skills and attributes

• Delta Lake, Unity Catalog and Databricks Workflows.
• Pipeline observability and alerting tooling (for example Databricks system tables, Datadog, Grafana or cloud-native monitoring).
• Data quality frameworks and automated testing for data pipelines.
• Experience ingesting varied source types: APIs, files, databases and streaming.
• Cloud platform experience (Azure, AWS or GCP).
• Comfortable working autonomously in a small squad, with clear communication to technical and non-technical stakeholders.
• A pragmatic, delivery-focused mindset suited to MVP work.