Database tradeoffs: Finding the sweet spot between performance and cos

With the exponential rise of data and most recently AI, organisations across industries face unique hurdles in leveraging it for actionable insights and undertake initiatives involving AI. From data sprawl to poor data quality to streamlining data integration across different data stores, scalability, price performance, TCO –there’s much to tackle.

To discuss the evolving database challenges, Couchbase, a distributed NoSQL developer database platform and YourStory recently organised a closed-room round table conference titled ‘Couchbase Unveiled: Minimising TCO, Maximising Performance’ in Bengaluru.

Participants spoke at length about their journeys and the unique roadblocks they faced in the management of databases. Some even shared insights around industry-specific issues that need different treatment. The idea was to share use cases and weigh in on ways to reduce the total cost of ownership (TCO) and maximise performance.

The round table participants featured Harish Rama Rao, Senior Vice President – Product Engineering, ACKO; Krishna BS, Head of Engineering, Dezerv; Nimesh Verma, CTO PL Division, Lendingkart; Rajesh Gupta, Director of Devops, Games24X7; Debarshi Roy, Vice President–Product, Credit Saison; Neelesh Kripalani, CTO, Clover Infotech; Pradeep Sreeram, Head of Product Engineering, Twid; Rishabh Jain, Head of Engineering, Vahan.ai; Mukesh Solanki, Head Infra and Devops, Kreditbee and Abhigyan Mandal, Head of Engineering, Virgio.

The roundtable was moderated by Rishabh Mansur, Head - Community, YourStory Media, and was guided by the expertise of Krishna Thirtha, Regional Business Head, Couchbase.

Key takeaways

The discussion kick-started with each speaker discussing their key challenges with some of the persisting fundamental challenges involving specific to their database landscape. For instance, unlike e-commerce or fintech, wealth management is completely devoid of the concept of an order. There's a well-defined workflow until it lands in a terminal state. The challenge is to model those constructs and the flow of money across various parts of investments without a customer ever placing an order.

There are also roadblocks regarding scale where the number of customers and transactions are less–however, the average ticket size and transaction value are high. For most companies, scale continues to be a recurring concern. An expert highlighted how the biggest issue for them is to tackle the sudden expansion and reduction of infrastructure as a consequence of seasonality.

Other companies brought to fore how a combination of high transactional value and scaling challenges is the culprit, which they are attempting to address by rebuilding the architecture to justify the high cost.

A key insight was highlighted during the meeting, which resonated with many leaders. For instance, some organisations deal with data of all types, for which they have multi-modal data storage. In that case, the challenges of TCO are different.

Speakers also examined the legitimacy of the same customer coming to them through different channels, balancing the vision of managing costs while hitting scale.

The discussion around challenges was followed up with actionable insights to overcome these issues. While some have been nipped in the bud, others are in progress.

For instance, in the absence of a good SQL administrator (owing to high prices or them being a rarity), the challenge is to manage compact database systems. Several companies have been partnering with third-party vendors who provide managed services or databases. Not only do they focus on database support but also influence organisations on the kind of technology that's fit for them in different use cases.

Application architecture is the key factor driving this change since it allows engineers to build on it. The long-term vision is for the tech experts to focus on this aspect since the other areas are more housekeeping-related.

Besides, a shift towards application-centric approaches and microservices has led to a simplification at a microservices level in terms of compute and store. Whether it's application logic or storage, companies try not to allow complexity to creep into any of the layers that include data. The biggest question here is: if the data modeling isn't complex, why does one need specialists?

With technology being the DNA of all businesses, there are also constraints around the cost. By outsourcing engineering work to vendors, it's easier for companies to build applications and focus on other important tasks that need their attention.

The biggest takeaway from the discussion was to look for ways to reduce the TCO, especially at a time when the churn is high.