The big data challenges facing companies today
Big data has become a key part of doing business, but that doesn’t mean that it’s easy. In the last five years 95% of businesses have undertaken a big data project, according to a survey of Fortune 1000 business leaders. However, the same survey revealed that less than half of those initiatives actually achieved measurable results.
There are a number of common challenges businesses face when it comes to big data, many of which come from the lack of big data experts internally. Companies know they should be using the information they own to inform and progress the business, but they lack the expertise to successfully implement projects.
Understanding
Often, the successful implementation of big data projects requires a degree of change across departments – be it in capturing more data, or changing how it is managed or shared with other teams. If some employees don’t understand the need for change or simply aren’t willing to play their part in bringing it about, then project progress is impeded. In order to minimise this kind of disruption, it’s important to take a ‘top-down’ approach. Once senior management have bought in to the concept, training and workshops can bring other employees up to speed on the importance of the project’s success to the future of the company.
Cost
Regardless of the approach you take to implementation of big data projects – on-premises or cloud solutions – the costs are high. On-premises solutions require a lot of hardware, which in turn requires space, electricity and security as well as administrators and developers to maintain it. While cloud solutions may be perceived as being lower cost, there is still a significant up-front investment needed for setting-up, configuring, training, maintenance and storage.
Of course, you’ll need to allow for future growth too, which will likely incur more expense. To avoid these costs spiralling and getting out of control, understanding the business’s big data requirements and having a clear and robust vision and strategy is essential.
Integration
It’s often the case that departments within a business work in data silos. Information is not readily shared across the whole company, partly due to the different platforms used by different teams not ‘’talking’ to one another.
Big data adoption is reliant on this not being the case. The key purpose of big data is to give one single 360 degree overview of the business; to do this information from every department needs to be accessible. This could mean a change in software for some departments or utilising APIs which overlay existing tools so that data can be shared. It will require a change in working processes for many employees – such as no longer using locally saved spreadsheets in order to make jobs ‘easier’.
Security
Security should be considered at the first stage of project planning. Not only because it is an opportunity for the business to review its existing security and policies (or lack thereof), but fundamentally because security should be built into the business’s big data solution from the very beginning.
A great example of this comes from the medical field; very sensitive patient data is stored all across the world, however the ability to analyse it all could provide great insights for those looking to cure diseases such as cancer. Security is a huge concern though, and in order to analyse the data it need to be decrypted, but now thanks to the use of fully homomorphic encryption, patient data could be analysed whilst still encrypted.
Big data is being collected constantly, but in order to make it us able and provide valuable insights which can drive the business forward, big data projects need to be implemented with a great deal of planning and expertise. However, the skills required to execute projects of this kind are in short supply. The University of York’s 100% online MSc Computer Science with Data Analytics course is designed for working professionals and graduates who want to start a career in this lucrative field. You’ll develop specialist skills and knowledge in machine learning, data analytics, data mining and text analysis via specialised modules and an independent data analytics project. You’ll also develop your core computer science skills such as computational thinking, computational problem solving and software development.
Our 100% online programmes allow you to study around work and home commitments, in your own time. Choose from six start dates per year, with a pay-per-module option available. You may be eligible for a government-backed postgraduate loan, covering the cost of the course.