Metadata Engineering and Privacy in Smart world


Machine Learning, AI and Bigdata : The Buzz word techs

While all hype, mostly justified, created around these technologies which has been heavily used by leading internet companies namely Google, Facebook etc is now being adopted or tested by every organization to explore how they could improve their operations, sales and of course cut some cost while achieve growth goals.

While these technology help organizations achieve better insight of the customer behavior and help predict their next move, it also raises some security and privacy concern too.

Everybody is Data scientist

Days are done when you need to have statistical masters or Phd and have to spend years learn sophisticated tools such as SAS to learn in and out of data science to run your models and make something out it. Computational evolution and dramatic cost reduction on data ownership, thanks to Bigdata on cheap hardware, is allowing pretty much every small to mid size firm to have their own data lake and run easy to use tools such as Trifacta and DataRobot to do advanced data analytics without even hiring a Data scientist.

Having that level of data power opens door for major data security concerns, as what if it goes into wrong hands?

Snowden Effect

Since Snowden blown the whistle about alleged mass surveillance by NSA powered by data from Internet and telecom companies, it created a tectonic shift in eyes of consumer how these companies are managing their data. After that incident almost every company revamped their privacy policy and you might have got multiple notification from them to read and acknowledge the new terms and condition and I could guarantee 99% people didn’t bother to read and simply accepted the annoying message to get rid of it for good.

Although it created more awareness among the users and changed the way companies store and use customer data, it hardly changed anything in terms of government policies or laws. Few eye washing laws might have been passed across western world but that is only to cover tip of the iceberg. Still increasing number of “concern” users still don’t trust organizations and government with their own data. I specifically used concerned word as not everybody care about their data. Most people simply, knowingly or unknowingly, made their whole life public by sharing everything on Facebook, Instagram and let the advertising vultures engineer with their data.

Now you must be hearing from all companies that we don’t share or sell your data to anybody. Now you can choose if you want to keep your data private or share.With new privacy policies the companies are not suppose or share or sell data without user consent.

We don’t store or share your data – Metadata Engineering to help

Now the internet companies do not need your data to analyze and predict your behaviour thanks for Metadata engineering. Take an example of new Photo application in iOS and Android, which automatically does the face detection, pet detection, location detection thanks to the new camera applications, which adds lot of metadata along with the image file to tell whole story behind the your photo.

Smartphone getting smarter

Lately you must be very happy with new iOS or Android update which “learns from you” and suggest you what is best option for you. For many people it is a very cool feature that lets Siri to recommend your schedule or next shopping item or next holiday destinations. Industry leaders may tag these features with any marketing cool names, but deep inside it is almost century old statistical analysis algorithms which are powering all these machine learning tools.

From a technology standpoint it is very interesting to learn and use these new machine learning tools to analyze and predict the next action, however, from privacy standpoint it is whole different story. Now the smartphone with their “learning feature” may upload all metadata to the server where it can be further refine and enriched for enhanced targeted marketing for products and same data could be used by Gov or criminal organization to for a totally different purpose.

The new normal

Whether you call it technological advancement or evolution, aforesaid data engineering are going to stay. As we are more connected then ever, in current fast moving world you need to stay updated to compete in the market. You can’t just switch off your smartphone and go to jungle it is just not possible. If I google right now how to keep your privacy, I see hundreds on blog post which tells how to protect it. I am not going to repeat them here as most of them either suggest switching off your smartphone or others are simply misinformed and ineffective solutions.

As a data engineer enthusiast myself, this could not be more exiting time that you have access to so many tools that lets explore and play with data, but I guess it won’t be fun to learn that how my personal data is being engineered to predict my next purchase.

On a lighter node, with continued advancement on automation and machine learning soon you will have your milk delivered to your door before it finishes, thanks for IoT sensors on your fridge and amazon echo.

If you are looking for getting end to end Metadata engineering solution powered by Bigdata and Machine learning please reach out to us now for free consultation.