New

Meteorological Data Engineer

Macquarie Global Services
United States, Minnesota, Minneapolis
Mar 13, 2025
We are a leading energy trading and investment firm currently expanding our weather analytics program to support high-stakes decisions in natural gas, power, and commodity markets. Our goal is to ingest, integrate, and continuously evaluate traditional numerical weather prediction models (GFS, HRRR, GEFS, ECMWF) as well as emerging AI-driven solutions (GraphCast, GenCast, NeuralGCM, and those to come). Leveraging the Pangeo ecosystem-including Xarray, Dask, and Zarr-we will efficiently store and process massive atmospheric datasets, making them readily available for real-time, customized analyses ensuring real-time, high-quality information for our traders. At Macquarie, our advantage is bringing together diverse people and empowering them to shape all kinds of possibilities. We are a global financial services group operating in 34 markets and with 55 years of unbroken profitability. You'll be part of a friendly and supportive team where everyone - no matter what role - contributes ideas and drives outcomes. What role will you play? Press space or enter keys to toggle section visibility As our Meteorological Data Engineer, you will architect and maintain the cloud-based ETL pipelines that drive our end-to-end weather data flow which includes real-time ingestion, transformations, and storage of traditional large-scale atmospheric datasets, as well as quick integration of newly emerging AI-driven forecasts. You'll interface with security, infrastructure, and DevOps teams to implement best practices, and you will work in tandem with the Meteorological Data Scientist to ensure the data is structured, validated, and readily accessible for advanced analytics and model evaluation. This will all support the team mission; enabling near-real-time availability of critical weather intelligence for high-impact trading decisions. What you offer Press space or enter keys to toggle section visibility 5+ years in data engineering, software development, or related fields with demonstrated success building ETL pipelines for large atmospheric or geospatial datasets Strong background using Dask or similar frameworks (Spark, Ray, Modin) for parallel compute, familiarity with AWS S3-based data ingestion, parallel I/O optimization, and containerization (Docker, ECS/EKS) Proficiency in workflow orchestration (DAGs, Airflow, Prefect) and DevOps (CI/CD pipelines) to deploy real-time data solutions Track record of tuning concurrency, chunk sizes, and data compression for near-real-time data pipelines Experience with the Pangeo stack of tools (specifically xarray, Dask, and Zarr) and familiarity handling typical meteorological data (i.e GRIB or netCDF files) a bonus, but not required. We love hearing from anyone inspired to build a better future with us, if you're excited about the role or working at Macquarie we encourage you to apply. About Commodities and Global Markets Press space or enter keys to toggle section visibility Commodities and Global Markets is a global business offering capital and financing, risk management, market access, physical execution and logistics solutions to its diverse client base across Commodities, Financial Markets and Asset Finance. Benefits Press space or enter keys to toggle section visibility Macquarie employees can access a wide range of benefits which, depending on eligibility criteria, include: Hybrid and flexible working arrangements One wellbeing leave day per year and minimum 25 days of annual leave Primary caregivers are eligible for 20 weeks paid leave along with 12 days of transition leave upon return to work and 6 weeks paid leave for secondary caregivers Paid volunteer leave and donation matching Range of benefits to support your physical, psychological and financial wellbeing Employee Assistance Program, a robust behavioural health network with counselling and coaching services Recognition and service awards Our commitment to diversity, equity and inclusion Press space or enter keys to toggle section visibility We are committed to providing a working environment that embraces diversity, equity and inclusion. As an inclusive employer, Macquarie provides equal opportunities to all individuals regardless of race, color, religion, sex, sexual orientation, national origin, age, disability, protected veteran status, genetic information, marital status, gender identity or any other impermissible criterion or circumstance. Our aim is to provide reasonable accommodations to individuals who may need support during the recruitment process and through working arrangements. If you require additional assistance, please let us know during the application process.