DBAs Face New Challenges: Trend in Database Administration

Introduction and Key Results

Data and information has long been recognized as companies' most valuable assets—and database administrators (DBAs) have been the key guardians of those assets. DBAs' primary responsibilities have been aimed at ensuring that data is safely, securely, accurately, and appropriately stored, managed, and maintained. DBAs must ascertain that organizations have the needed capacity to meet their data needs; that databases perform at optimal levels, and that databases function properly and are backed up. If problems arise, DBAs are the ones called on to troubleshoot and resolve the issues effectively and efficiently. And if a system fails for whatever reason, the DBAs are responsible for recovering the data stored in their databases.

And those are just the baseline responsibilities for most DBAs. Over the last 10 years, major trends in technology have shaped and reshaped the ongoing role of the DBA in many organizations. New data-producing devices—ranging from smartphone technology to the Internet of Things—have continued to drive data growth, often requiring DBAs to manage both more database instances as well as a wider range of database management systems. New data types coupled with emerging applications have led to the growth of non-relational data management systems. Frequently, the management of non-relational data platforms such as NoSQL databases has fallen under the purview of the DBA. Cloud technology has enabled enterprises to move some data off-premises, complicating the overall data infrastructure. And new database management tools have allowed some common functions to be more fully automated.

This shifting environment has led to an expansion of the DBA role in some settings. DBAs are looked to for advice and guidance in developing the overall data management infrastructure, including when to move data to the cloud. Moreover, with the growth of the DevOps approach to application development, many DBAs are more deeply involved with application and database development.

To gain insight into the evolving challenges for DBAs, Quest commissioned Unisphere Research, a division of Information Today, Inc., to survey DBAs and those responsible for the management of the corporate data management infrastructure. The more than 200 respondents came from a wide range of industries and companies of different sizes. More than half were DBAs or database developers and 20% came from companies with more than 5,000 employees. Details about the respondent pool can be found in the appendix.

Among the key findings are:

  • Structured data under management continues to grow at a substantial rate, requiring DBAs to manage both more database instances and a wider variety of relational database management systems.
  • Cloud technology plays a significant role in hosting databases and that role will continue to grow. Cloud approaches will have the greatest impact on database administration over the next three years.
  • NoSQL platforms represent a smaller but important slice of the data management infrastructure.
  • The number of DBAs that organizations need is holding steady.
  • Database management tools have reduced the time some DBAs must invest in some routine operations, allowing them to expand their roles in other areas. Database performance continues to be the top priority for DBAs.
  • Many DBAs are increasingly involved in application development.
  • DBAs are emerging as key influencers in the overall data management infrastructure.

The Data Infrastructure

Although the emergence of a wide variety of new and significant data types that have been brought under management during the last decade has captured a lot of attention, managing structured data continues to be central to most organizations. Companies are managing massive amounts of data and support an impressive number of database instances. As Figure 1 shows, nearly 60% of the respondents have more than 100TB of structured data under management. And, that data is spread over a lot of database instances. More than 40% of the respondents have more than 100 database instances running. Close to 20% have more than 500 database instances in operation. (See Figure 2.)

Figure 1: Approximately what is the total amount of structured data under management in your ON-PREMISES relational database management systems?

Figure 2: Approximately how many relational database instances do you have running in your organization?

To manage this data, companies use an array of database systems. As Figure 3 demonstrates, Oracle and Microsoft SQL Server continue to be the overwhelming market leaders in relational database management systems (RDBMSs). But respondents also mentioned using more than 20 relational, NoSQL, or MultiValue DBMSs in their operations.

Figure 3: Which relational database management system platforms/brands do you have running in your organization on-premises?

The amount of data under management and the number of database management systems in use has several implications for database administrators. As can be inferred from Figure 3, many companies support more than one database management system. Indeed, as Figure 4 reveals, 40% of the respondents support four or more brands of database management systems.

Figure 4: Including both relational and non-relational database management platforms, approximately how many different brands of database management systems does your organization support?

At the same time, most DBAs are responsible for managing multiple database instances, with 10% of the respondents indicated that DBAs in their organization manage more than 100 database instances. (See Figure 5.)

Figure 5: Approximately how many database instances does each DBA manage?

Finally, the rate of growth of structured data under management remains impressive. As Figure 6 shows, around 90% of the survey respondents indicated that structured data in their on-premises databases was growing at an annual clip of 50% or less. Obviously, a 50% growth is significant, but with so many potential data- generating devices coming on stream, perhaps not unanticipated.

Figure 6: How fast is the volume of data under management in your ON-PREMISES relational database management systems growing annually?

This aspect of the picture for data management and database administration is not entirely new. A survey conducted three years ago produced similar results with 10% of the respondents reporting that DBAs in their organization supported more than 100 database instances and around 40% indicating that each DBA was responsible for at least 25 database instances. Interestingly, in that survey, respondents felt the number of database instances each DBA was tasked with managing was increasing. That growth may have taken place to a degree but the overall landscape has not radically changed. At the same time, three years ago, DBAs were also routinely required to support database systems from several different vendors. Even the rate of structured data on-premises has been relatively stable over the past several years, with a majority of companies seeing structured data growing between 10% and 50% annually.

While the data growth rate, number of databases instances, and number of platforms that each DBA must support has not dramatically changed in the last three years, the database infrastructure has gotten more complex. Two factors have driven increased complexity. First, the cloud has emerged as a significant platform for database management. Second, non-relational databases, including NoSQL databases, have gained a notable foothold in the corporate environment.

Databases in The Cloud

Slightly less than half of the respondents currently support at least some databases in the cloud. Of those who have not yet begun to support databases in the cloud, only around onethird of respondents have no plans to do so. Another third of respondents anticipate using databases in the cloud within the year. The final third have plans to move to the cloud but not within the year.

Managing relational databases in the cloud is clearly an emerging technology. As Figure 7 illustrates, slightly more than two-thirds of the respondents have supported relational databases in the cloud for two years or less.

Figure 7: How long has your organization managed relational databases in the cloud?

Those companies moved to the cloud for a variety of reasons. About one-third were primarily seeking increased flexibility, while around 30% were hoping for lower costs. Interestingly, only about 20% moved to the cloud to support a new use case, which is often the reason for integrating a new RDBMS into the overall infrastructure. (See Figure 8.)

Figure 8: What was the primary reason you chose to manage at least some structured relational data in the cloud?

As Figure 9 shows, the organizations that have moved to the cloud are managing a significant amount of data there. The move to the cloud is not a marginal or trivial element of their data management environment. On the other hand, in general, onpremises database systems in most cases still account for most of the data under management. (See Figure 10.)

Figure 9: Approximately how much relational data does your organization manage in the cloud?

Figure 10: Of all the structured data under management in your organization, approximately what percentage is managed in the cloud?

Somewhat unexpectedly for a new technology, in general, the rate of growth of data stored on DBMSs in the cloud is about equal to the rate of growth of DBMSs on premises. However, about 15% of the respondents indicated that their data in the cloud is growing more than 50% a year and more than 5% said it is growing more than 100% annually. (See Figure 11.)

Figure 11: How fast is the volume of data under management in your cloudbased relational database management systems growing annually?

About half of the respondents indicated that they used a hybrid cloud infrastructure with the rest split between using public cloud services and establishing a private cloud. Amazon Web Services hold the largest share among the respondents to this survey with nearly 40% saying they use AWS. About a quarter of the respondents use Microsoft Azure for their cloud platform and a similar number use Oracle Cloud. The remainder is divided among a wide variety of vendors including EnterpriseDB, Rackspace, SAP HANA, Google Cloud, and others

Growth of NoSQL

The term "NoSQL" represents a group of technologies that use non-relational approaches to data management. In general, NoSQL provides a mechanism for storing and retrieving data that does not use the tabular format associated with relational databases. While non-relational data management systems have been on the market for more than half a century, the demands of Web 2.0 applications have driven significant interest in NoSQL approaches. In some cases it is thought that a NoSQL solution will scale better and in others it may provide more efficient operation for specific applications.

Around one-third of the survey respondents are currently supporting NoSQL data management systems. Of those that don't, a little more than a quarter (27%) plan to move to NoSQL for some applications within a year. A little under half (44%) have no plans to use NoSQL for the foreseeable future. And, similar to cloud adoption, NoSQL is still a new technology for most enterprises. About two-thirds of the respondents have been using

NoSQL platforms for less than two years. (See Figure 12.)

Figure 12: How long has your organization managed NoSQL databases?

As Figure 13 shows, organizations turn to NoSQL solutions for specific reasons—generally to support a new use case or because they see the need to manage semi-structured or unstructured data. And, so far, NoSQL solutions are deployed only for a defined slice of the overall enterprise data. (See Figure 14.) Nevertheless, NoSQL databases frequently manage a significant amount of data and, in some cases, a very significant amount, as shown in Figure 15.

Figure 13: What was the PRIMARY reason you chose to manage at least some data with a NoSQL database platform?

Figure 14: Of all the data under management in your organization, approximately what percentage is managed in NoSQL databases?

Figure 15: Approximately how much data does your organization manage with NoSQL databases?

The volume of data under management using NoSQL platforms is growing steadily, though perhaps not quite as fast as the volume of data under management using RDBMSs. (See Figure 16.) Among the survey respondents, MongoDB was the most popular platform, followed by Cassandra and Redis.

Figure 16: How fast is the volume of data under management in your organization's NoSQL database management systems growing annually?

Taking a big picture view, it is clear that most organizations support a heterogeneous database environment. Enterprises need multiple database platforms for many reasons as Figure 17 demonstrates. The most important reason is that different platforms are needed to support a variety of applications and use cases. But multiple database platforms are also required for a host of other reasons ranging from a desire to avoid vendor lock-in to rogue database developers simply moving in their own direction.

Figure 17: What are the most important reasons for using multiple database platforms in your organization? (Multiple responses allowed.)

Role of the DBA

As could be expected, as the database environment changes, so does the role of DBAs. About 25% of the organizations represented in the survey have 25 or more people holding the title of DBA. Slightly less than half (46%) have one to 10 DBAs. As a result, the survey paints a comprehensive picture of the role of the DBA in both larger and smaller companies.

The movement of databases to the cloud coupled with increased automation of the functions historically associated with database administration has led to some speculation that companies may be able make do with fewer DBAs in the future. That does not seem to be the case—at least, not yet. Indeed, more than 60% of the respondents said the number of people with the title "DBA" was holding steady, while around 20% said the number of people holding that role was increasing.

As noted earlier in Figure 5, individual DBAs manage multiple database instances. Around 70% of the respondents indicated that the DBAs in their organizations managed 11 or more database instances. Around 10% said their DBAs managed more than 100 database instances each. And as Figure 18 shows, in most settings, the number of database instances individual DBAs manage is increasing.

Figure 18: Is the number of database instances for which each DBA is responsible increasing, decreasing, or staying the same?

Individual DBAs are responsible for managing multiple database management platforms, as well. About 50% of the respondents say that DBAs in their enterprises typically manage database management systems from two different vendors. A little more than a quarter (26%), said their DBAs manage platforms from three to five vendors.

As Figure 19 illustrates, in most cases, the same DBAs are responsible for administering on-premises databases as well as those databases that are hosted in the cloud. In those cases in which DBAs do not have responsibility for databases in the cloud, that responsibility is assumed by a range of other IT personnel, including dedicated teams, DevOps groups, network administrators, and new product groups. In some cases, the responsibility is outsourced either to the vendor or a third party.

Figure 19: Are the same DBAs responsible for managing relational database management systems on-premises also responsible for managing relational database systems in the cloud?

To a large degree, current DBAs are also being charged with the responsibility of managing NoSQL databases, as well. (See Figure 20.) In cases where they are not, NoSQL databases are often managed by the applications Ops teams or new product teams.

Figure 20: Are the same DBAs responsible for managing relational database management systems also responsible for managing non-relational systems as well as NoSQL databases?

Clearly, as new data management approaches like NoSQL and database deployment in the cloud are implemented in many enterprises, the responsibility for managing them fall to the current DBA staff. That said, the exact priorities for managing databases in the cloud, while similar, are not exactly the same as managing databases on-premises. As Figure 21 shows, the top priority for managing on-premises is ensuring performance, followed by maintenance and configuration. In contrast, while performance remains the top priority for databases in the cloud, capacity planning and security jump in importance. (See Figure 22.)

Figure 21: What are the most important reasons for using multiple database platforms in your organization? (Multiple responses allowed.)

Figure 22: What do you consider to be the most important responsibilities for DBAs in managing databases in the cloud? (Multiple responses allowed.)

As could be anticipated, along with changing DBAs' priorities, moving databases to the cloud also has an impact on the time the DBAs spend performing different tasks. Figure 23 shows that DBAs devoted most of their time for on-premise databases to maintenance, performance, and configuration management. For DBAs managing on-premises databases, while performance is the top priority, maintenance is the most time-consuming task.

Figure 23: To which responsibilities do your DBAs who are administering on-premises database typically devote the most time? (Multiple responses allowed.)

In contrast, Figure 24 illustrates the tasks to which DBAs who are managing databases in the cloud devote the most time. As could be assumed, the time devoted to maintenance plunges, as does the time devoted to configuration management. Instead, DBAs are much more focused on database performance. In other words, moving to the cloud apparently allows DBAs to focus more fully on their top priority. Attention to security also significantly climbs the priority list in many enterprises.

Figure 24: To which responsibilities do your DBAs who are administering databases in the cloud typically devote the most time? (Multiple responses allowed.)

Clearly, moving databases to the cloud has an impact on many DBAs' priorities and the attention they devote to different tasks and functions. DBAs are active in the management of databases running in the cloud but they can focus more intently on their top priority—database performance. As one respondent put it, the only change is that DBAs are now responsible for the performance of databases not under their control.

In fact, the move to the cloud has an impact on DBAs in other ways too. In some cases, DBAs must become familiar with the tools provided by the cloud vendor. Operating systems and network issues fall less frequently to the DBA. But perhaps most importantly, in many cases, since the routine and mundane elements in administering databases can be eliminated, DBAs will be free to focus on higher value activities.

The Changing Tasks of the DBA

The growth of databases in the cloud and NoSQL technology necessarily has led to changes in the way that DBAs spend their time. Automation is also a factor in changing the way DBAs go about their responsibilities, as RDBMS vendors, database tools vendors, and cloud services provide vehicles for DBAs to more effectively perform their duties. Figure 25 presents an array of tasks that DBAs generally perform. Consistent with other findings in the survey, performance-tuning is consuming more of the DBAs' time, while tasks like the verification of backups is taking less time. The responsibilities that are being automated most successfully are illustrated in Figure 26. Verifying backups and that all database instances are operating are the most automated functions. Performance-tuning and cloning and provisioning databases for test and development require the most attention.

Figure 25: Is the amount of time you devote to each responsibility increasing, decreasing, or staying the same?

Figure 26: Which of these tasks are automated, performed manually, or not generally performed?

DBAs' day-to-day responsibilities are clearly evolving. In some organizations, DBAs have a more integrated role in application development, particularly in companies that have adopted a DevOps framework. In addition, many DBAs are playing increasingly important roles in advising both IT and C-level management about the major issues in the data infrastructure.

About half of the respondents in the survey indicated that the DBAs in their organizations are involved in the application development process (53%). And while respondents indicated that in most cases (52%) senior IT management, such as CTOs or CIOs, initiate projects to utilize cloud platforms and also give final approval for such projects (60%), DBAs are deeply involved in those kinds of efforts. As Figure 27 shows, DBAs are involved in researching potential solutions, preparing solutions, and then, of course, maintaining the chosen solutions

Figure 27: In which parts of the process of deciding to maintain databases in the cloud are DBAs in your organization involved?

Looking Forward

While there are many exciting new technologies and challenges emerging in data management, over the next three years, the ongoing utilization of cloud platforms is poised to command the most attention. As Figure 28 shows, in terms of overall data management, the move to the cloud was tabbed by around 50% as the top challenge for DBAs in the near future, followed by increased automation (41%). Along the same lines, the cloud was overwhelmingly tapped as the technology trend that will have the most impact on database administration over the next three years. (See Figure 29.)

Figure 28: In terms of the data management infrastructure in your organization, over the next three years, what do you think will be the top challenges facing DBAs?

Figure 29: Which technology trends do you think will have the most impact on database administration over the next three years?

For the working DBA, however, the biggest challenge over the next couple of years is one with which most DBAs are very familiar—how to manage more databases per administrator. (See Figure 30.)

Figure 30: In terms of database administration itself, over the next three years, what do you think will be the top challenges facing DBAs? (Multiple responses allowed.)

Conclusion

Along with the data management infrastructure itself, the role of the DBA continues to evolve. Some of the changes reflect trends that have been in place for a long time and remain unabated. The amount of structured data under management is maintaining an impressive growth rate, for example. In response, individual DBAs are responsible for administering more database instances and supporting platforms from multiple vendors. However, certain more routine tasks associated with database management are being effectively automated, allowing DBAs to devote less time to them.

Some trends are newer. Both cloud and NoSQL approaches are emerging as significant elements in data management for many organizations. DBAs are frequently being asked to manage those technologies, as well as traditional, on-premises relational databases. Supporting a database in the cloud often leads to a major shuffling in a DBA's day-to-day tasks and functions, freeing up space to address new duties and challenges.

In many organizations, DBAs have been integrated into the application development process earlier than before. Along the same lines, while they are not the decision makers, DBAs frequently have a significant amount of influence in shaping the ways that the enterprise incorporates new technologies into its overall data management infrastructure. In short, the two major trends—the move to the cloud and the increased use of automation—are enabling DBAs to provide higher value services to their organizations.

Appendix: Demographics

Figure 31: What is your primary job title?

Figure 32: What is your organization's primary industry classification?

Figure 33: Approximately how many employees (including all locations, branches, and subsidiaries) are in your in your organization?