Data Engineer

Data Engineer About the Role

We are seeking a motivated and fast-learning Data Engineer to join a growing London-based data team focused on building and maintaining a large-scale market intelligence platform.

This is a hands-on engineering role where you will design, develop, and maintain ETL pipelines, web scrapers, and transformation workflows that process millions of data points daily across multiple regions.

You will work closely with data and engineering colleagues to help shape scalable data infrastructure and contribute to innovative AI-driven workflows.

The organisation operates a hybrid working model, with team members typically attending the London office approximately three days per week. Visa sponsorship may be considered for suitable candidates.

Key Responsibilities

• Design, build, and maintain reliable and scalable data pipelines using Python and cloud technologies.
• Develop and maintain Python-based web scraping solutions using tools such as Playwright and BeautifulSoup.
• Write and optimise SQL transformation workflows to convert raw data into analytics-ready datasets.
• Build and maintain LLM-driven workflows within ETL pipelines, including validation and quality checks.
• Implement data quality, testing, and validation frameworks across extraction and transformation processes.
• Collaborate with cross-functional teams to support data-driven decision-making.
• Stay up to date with developments and best practices in data engineering and AI-enabled workflows.

Required Skills & Experience

• Master s degree in Computer Science, Engineering, Data Science, or a related discipline.
• Minimum 3 years experience in a Data Engineering or related role.
• Strong Python development skills, including object-oriented programming and production ETL/ELT pipeline development.
• Strong SQL skills with experience writing and optimising advanced queries.
• Hands-on experience with cloud data platforms such as GCP, AWS, or Azure.
• Experience working with web scraping frameworks such as Playwright, Scrapy, or similar.
• Familiarity with LLM APIs and integrating AI capabilities into data workflows.
• Experience with Git, Linux, Docker, and CI/CD workflows.
• Experience working with NoSQL databases.
• Strong communication and problem-solving skills.
• Experience with data testing and validation frameworks.

Desirable Skills

• Experience using AI-assisted development tools and coding assistants.
• Knowledge of Infrastructure as Code tools such as Terraform.
• Experience with data visualisation platforms such as Tableau, Power BI, or Looker.
• Familiarity with machine learning or data science concepts.
• Cloud platform certifications.
• Experience working within agile, fast-paced environments.

Technical Environment Languages & Libraries

• Python
• SQL
• YAML

Cloud & Data Technologies

• BigQuery
• Cloud Storage
• Cloud Functions / Cloud Run
• Firestore
• Dataflow
• Dataform
• Compute Engine
• Secret Management tools

DevOps & Tooling

• GitHub
• GitHub Actions
• Docker
• Terraform
• Linux
• Pytest
• Jira

Equal Opportunities

We are committed to creating an inclusive and diverse workplace and welcome applications from candidates of all backgrounds and experiences.