AUSTIN, Texas & CAMBRIDGE, Mass.--(BUSINESS WIRE)--A survey commissioned by data.world and DataKitchen reveals a disturbing state of affairs among data engineering professionals. The study of 600 data engineers, conducted by Wakefield Research, suggests an overwhelming majority are burned out and calling for relief.
- 97% report experiencing burnout in their day-to-day jobs.
- 70% say they are likely to leave their current company for another data engineering job in the next 12 months.
- 79% have considered leaving the industry entirely.
Among the highest compensated data professionals, a data engineer prepares data for data analysts, scientists, and people that use self-service BI tools to perform the analytics that guide data-driven decisions. Turnover in this critical role is especially disruptive for companies given the significant influence of data engineers on data productivity and overall business agility.
The data engineers surveyed identified significant sources of burnout including:
- Spending too much time finding and fixing errors
- Manual, repetitive processes related to data prep and pipelines
- Relentless pace of requests from colleagues
Other challenges cited by data engineers include:
- 91% report frequently receiving requests for analytics with unrealistic or unreasonable expectations.
- 87% say they are blamed when things go wrong.
- 69% say their company’s data governance policies make their day-to-day job more difficult.
- 89% report frequent disruptions to work-life balance due to unplanned work.
These factors force data engineers to work long, irregular schedules that take a toll on their well-being. In fact, 78% of survey respondents wish their job came with a therapist to help manage work-related stress.
“Companies are hiring data professionals to work on complex problems with little attention to designing the workflows that enable efficient team collaboration,” said Christopher Bergh, CEO, Co-Founder and Head Chef at DataKitchen. “It’s basically setting an entire, highly skilled class of workers up for failure and then blaming those same workers when projects fail. We hope to see a shift in the importance of process and workflow design in managing teams of data professionals.”
The problems in data organizations may seem deeply entrenched, but data engineers identified new methodologies like DataOps to address systemic inefficiencies. For example, 78% of engineers and 91% of managers stated that DataOps is very important or essential to incorporate to their data practices.
DataOps, and its counterpart Agile Data Governance, automate end-to-end data lifecycle workflows, including error mitigation, development, deployment, data discovery, observability, and governance. By automating these tasks, data organizations can address the causes of data engineer burnout while improving the quality, agility, veracity, and usability of analytics.
“Our survey serves as a wake-up call to data organizations,” said Bryon Jacob, CTO and Co-Founder of data.world. “Data engineers are calling for change. That means instituting reliable, efficient and repeatable workflow processes that will improve analytics collaboration and productivity while restoring sanity to the lives of data engineers.”
Additional resources
- DataKitchen and data.world will host a joint webinar on Wednesday, October 27 at 12:00 PM ET to go into great detail on the sources and mitigations of data engineer burnout. Register here.
- Download the white paper for an in-depth analysis of the survey results.
- View and analyze the raw survey data on data.world.
About data.world
data.world is the enterprise data catalog for the modern data stack. Our cloud-native SaaS platform combines a consumer-grade user experience with a powerful knowledge graph to deliver enhanced data discovery, agile data governance, and actionable insights. data.world is a Certified B Corporation and public benefit corporation and home to the world’s largest collaborative open data community with more than 1.3 million members. Our company has close to 50 patents and has been named one of Austin’s Best Places to Work six years in a row. Follow us on LinkedIn, Twitter, and Facebook, or join us.
About DataKitchen
DataKitchen offers the only complete enterprise DataOps Platform that enables organizations to quickly, intuitively, and successfully implement and manage an end-to-end DataOps program using tools they already own. The Platform serves as the command center for a DataOps program. It simplifies complex toolchains, environments, and teams so that the entire data analytics organization can quickly innovate, seamlessly collaborate, and instantly deliver the kind of error-free, on-demand insight that leads to one successful business decision after another.
