Another aspect of this I've observed -- Personal sense of value (and industry pay goes into this) contributes to partitioning of work. If we're charitable, it's from a belief of comparative advantage, and if we are brutally honest about some people, it's because people often feel that "_____ isn't a good use of their time." This is also fed by the "sexiest job of the 21-st century" saying that's been created.
We see this in data science and machine learning where people complain about spending their time cleaning data, etc... when their time should be spent "generating insights/etc." We also see that those insights are interesting but not very useful if they aren't actionable, too costly or too impractical to implement.
Ultimate value is related to being able to contribute to and achieve the holistic outcome, but the lens of success is often focused on models or insights instead. This is a cultural and organizational problem, rather than a technological one. It also takes a dose of humility to appreciate the true value of the so-called dirty work.
I see this with my own work. I maintain the Elasticsearch Learning to Rank plugin. People assume it's all magic machine learning. The reality is much of the work involves understanding Elasticsearch plugins, informed by machine learning that needs to happen. Oh and 50% of the work is support and fun things like Maven repos :)
Another point is that while we technologists love to marvel at data science and machine learning, it still begs the question of what value does it bring to the business. Does the added responsibility of creating all the infrastructure and processes worth it to justify a 5% increase in conversion rates? As you say, even the dirty work has a cost and whether that cost is worth paying to find out that there's nothing you can do to improve the business. That's why all the massive multi-year central data warehouse cleansing type projects keep failing without yielding much value. There's just a lack of focus on delivering incremental value with these data projects.
I think currently it creates new possibilities to do business. Computer vision is that point that nowadays it is more engineering than data science so adding something like somewhat good object detection is not that hard. NLP is probably same point where cv was 5 years ago so we start seeing very good NLP models.
Definitely. Even as an engineer working on CV 10 years ago, the hard part wasn’t object detection but rather network bandwidth to stream incredible amounts of data and processing it in real-time.
Spoke to an experienced engineer who used to lead NLP at MSFT and same comment. NLP models are already fantastic and it isn’t very hard to build a smart chatbot. The implementations these days are just very poor because they are not well thought out from a user perspective.
We see this in data science and machine learning where people complain about spending their time cleaning data, etc... when their time should be spent "generating insights/etc." We also see that those insights are interesting but not very useful if they aren't actionable, too costly or too impractical to implement.
Ultimate value is related to being able to contribute to and achieve the holistic outcome, but the lens of success is often focused on models or insights instead. This is a cultural and organizational problem, rather than a technological one. It also takes a dose of humility to appreciate the true value of the so-called dirty work.