Infographic: Data Engineering Evolved
Ascend.io, the data engineering company, announced results from a new research study about the work conditions of data scientists, data engineers, and enterprise architects in the U.S. Conducted in June 2020, findings from over 300 professionals reveal key insights on their teams’ current workload, productivity bottlenecks, and perspectives on automation and low-code technologies. Since the onset of the COVID-19 pandemic, the majority (78%) of data professionals have been asked to take on responsibilities outside of their core job function, with 97% now signaling their teams are at or over capacity. To support their teams and increase overall capacity, 89% of data professionals are turning to automation, low-code, or no-code technologies, with 73% citing automation as an opportunity for career advancement.
Pressures on Data Teams Escalate
The overwhelming majority of data teams are currently at or over work capacity, with just 3% citing they have extra capacity for new projects. Moreover, data executives (including VPs and directors) are nearly two times more likely to indicate their teams are overloaded than team leads and individual contributors themselves, signaling a significant backlog and increasing demand by the business. When asked which team was the most backlogged, enterprise architects led the way, followed by data engineers. However, respondents were over 3.5 times more likely to identify their own team as the most backlogged over others. Despite pointing to their own team as the backlog, the research found compelling patterns emerging around the need for data engineering resources to complete their work.
All Roads Lead to Data Engineering
Slow iteration cycles in data teams lead to significant delays for downstream users and decision makers, hindering teams’ ability to meet the data needs of the business. The delays vary by role: data scientists are mostly impacted by having to ask others for access to data or systems (48%), whereas data engineers are mostly held back by maintenance of existing and legacy systems (54%).
Organizations are quickly discovering that data engineers are essential to unlocking the value of data and to removing bottlenecks across the entire data team,” said Sean Knapp, CEO and founder of Ascend.io. “LinkedIn’s 2020 Emerging Jobs Report found that data engineering has surged onto the scene, quickly becoming one of the top-ten jobs experiencing tremendous growth. At present, there are simply not enough data engineers to meet the demand. To enable more data professionals to tackle the growing backlog of data engineering tasks, tools such as automation, low-, and no-code technology can provide tremendous leverage and scalability for existing data engineers, while at the same time enabling a new era of citizen data engineers.
Data Teams Level Up With Automation to Advance Careers and Empower Teams
Data professionals are increasingly turning to automated solutions to solve their team’s current challenges. 62% of respondents plan to implement automation technology to increase their team’s bandwidth, followed by buying new products or tools (47%), replatforming and retiring legacy technologies (41%), and hiring more staff (41%).
The majority (73%) of respondents view automation as an opportunity to advance their career.
Enterprise architects are the most welcoming of automation, with 85% citing that it would provide an opportunity to further advance their career, followed by data engineers at 72%.
Perspectives about what is holding data teams back are more diverse depending on who was asked:
- 77% of financial services respondents plan to implement automation to increase their team’s bandwidth, compared to the overall average of 62%. The more competitive nature of some sectors is perhaps driving faster adoption of data technology to drive business results than in others.
- Meanwhile, 75% of healthcare respondents indicated plans to replatform and retire legacy technologies, compared to the overall average of 41%. Some industries may still need to modernize their overall technology stack before data teams can have a greater impact on the business.
More Than Half of Data Engineers Seek Low- and No-Code Technologies
The survey results indicate that across the spectrum of startups to large enterprises, data teams today are looking to low- and no-code solutions to accelerate innovation and productivity. More data professionals are seeing the value of these tools, with 80% of respondents already using or considering them to support their team.
25% of data engineers are using low- and no-code solutions today and 53% are currently considering such technology, compared to 48% of enterprise architects and 42% of data scientists.
The research demonstrates that across the board, teams are being asked to do more with less and they are turning to low-code solutions as a means of continuing to deliver on business needs,” continued Knapp. “Low-code solutions are great at simplifying what used to be incredibly complex tasks. They also tend to encourage better modularity and system design, which keeps things simpler and more maintainable.
Originally posted by insideBIGDATA
Found this article interesting? You might want to check out this one too!
Are you a digital practitioner who has got more questions than answers during these uncertain times? Contact us submitting a quick form here. We are here to support you!
Digital analytics, cloud, data science, programmatic or optimisation expert, and looking for a job at the moment? Check out our latest live vacancies here
