Data Engineer

Data Engineer

Your Role and Responsibilities

The Weather Company, an IBM Business is seeking a Data Engineer for our NYC office. This role will require working with both technical and business partners to manage data and support data science and machine learning efforts.

Some responsibilities include:

  • Develop large-scale data structures and pipelines to organize, collect, transform and standardize data from multiple fragmented source in various formats to generate insights and address reporting needs.
  • Map data sources, manage metadata, and focus on data governance and validation to ensure we have standardized data and key metrics for the business.
  • Identify and resolve data issues to ensure the quality and consistency of data; determine and implement opportunities for automation.
  • Evaluate existing and develop new metrics, reports, and dashboards for the business as necessary.
  • Communicate data processes and insights through visualization and presentations.
  • Work with cross-functional teams to gather reporting and analysis needs.
  • Support data-driven decision making across the company and participate in overall company data strategies; explore additional data sources and analytics tools when necessary.
  • Support ad hoc analysis requests from various audiences.
  • Determine/create opportunities for advanced analytics and prediction.


Required Professional and Technical Expertise
  • 3 Years of professional experience, at the minimum, in data engineering or programming.
  • Manage the processing of Big Data using Python/PySpark, R, Hive, SQL, shell scripting, etc.
  • Develop API data collection jobs, cleanse, transform, join, and adjust data sources, and eventually store organized and ready-for-use data sets in enterprise data lakes.
  • Develop data pipelines, automated workflows, and analytical processing solutions in applicable Big Data and Hadoop ecosystems; strong conceptual knowledge of distributed frameworks and Hadoop architecture components.
  • Work in structured or unstructured databases, such as AWS RedShift, AWS S3, MongoDB, Hive, etc., to develop data models, write and optimize queries, and manage complex data sets
  • Use Source Code and Version Control systems like Git, etc.
  • Comfortable working in a hybrid cloud environment, deploying resources across local, remote and large-scale distributed computing platforms.

Preferred Professional and Technical Expertise
  • Experience in engineering data in the digital advertising industry; familiar with data sources from Google Ad Manager Data Transfer, API, and UI.
  • Familiar with BI and data visualization in open sourced environments like Dash/Plotly, or with commercialized packaged tools like Qlik, Tableau, etc.
  • Familiar with open sourced tools and packages that can be introduced to streamline automated workflows
  • Good communication and presentation skills to explain technical solutions to either technical peers or non-technical audiences

Location Statement

Being You @ IBM
IBM is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.

Back to top