Cloud technologies are changing the way we operate and do business. They enable us to focus the time and energy of resources on developing applications rather than spending time maintaining infrastructure. While cloud technologies enable business to take the leap, it is vital to understand how these technologies impact organizations and change organization models.
This article talks about the changing structure of analytics organizations in the enterprise, which traditionally had fully scaled analytics teams. These large teams in large organizations have many aliases — “Data team”, “BI team”, “Central data organization”, “BI CoE” — and consisted of multiple roles with clearly defined responsibilities. In a typical organization you would see roles such as ETL Developer, BI Developer, BI Server Admin, QA, Data Modeler, DBA, Data Analyst, and Data Scientist. All these roles had a clear line of responsibilities but caused overheads because of increased hand-offs and too many points of failure from shared responsibility.
Along with the technology shift, it has become almost vital to look toward newer organization models that keep the focus on business outcomes and reduce overheads. This aligns with the concept of Zero Based Thinking.
Zero-basing organizations, which use zero-based principles as a lens to reshape organizational structure and operations, can unleash greater productivity. The resulting purpose-built enterprise ensures that staff and resources are allocated to the highest-value areas of the business. — McKinsey
Organizations need leaner and more agile structures to focus on business outcomes. Following are the key roles that should be part of an analytics organization focused solely on delivering business value.
Data Engineer
This role has been talked about almost everywhere and is the anchor of the complete team. This is not just an ETL Developer, a DBA, or a report writer — it is a mix of all these roles. Data Engineers are responsible for setting up the relevant infrastructure, provisioning resources, setting up access, sourcing data, manipulating data, wrangling data, making data available to end users, and helping organizations consume data. In short, whenever data is accessed, manipulated, or written, it is handled by data engineers.
Data Scientist
This is a specialized skill that every data-driven organization needs. Data scientists consume the data wrangled by data engineers and use complex equations and statistics to write models. These models understand patterns in the data and can predict the next expected outcome with a certain confidence level. This is the role that takes analytics to higher maturity levels.
AI Engineer
Not many people have talked about this role, but I feel it is going to be the next important role in the organization. There is huge advancement in the field of commercial AI (COTS AI) services such as speech recognition, image recognition APIs, text analytics, search, and bots. These commercial AI products enable rapid application creation and use of pre-built, pre-trained ML models for prediction.
With the advent of self-service BI tools such as Power BI and Tableau, report writing has moved to business users. Maintenance roles have slowly started disappearing because of cloud. QA skills will be part of development teams because of the move towards DevOps. The three roles above will provide all key skills for an analytics team to succeed.