Weava Collection - Research on AI and the crowd economy (data, algorithm, machines, tasks, intelligence)
- Even though the rise of artificial intelligence means that the Turkers are the ones coaching the machines, workers still feel subsumed by the system. Ms Milland dreams of one day starting her own platform for tasks, a worker-owned platform that would treat workers better. “If we can’t have a union, we can have a platform, a co-operative crowdsource platform,” she says. “That’s the only way we can hope to make a difference.”
- One of the chores Mr Bhatia worked on through Mechanical Turk involved identifying “pins” for Pinterest, the online pinboard. He would be shown a photo — or pin — and then choose other pins that were similar to it, enabling Pinterest’s artificial intelligence engines to get better at predicting the pins a user will like.
“We are the intelligence behind that artificial intelligence,” Mr Bhatia says with a hint of pride. “I feel excited to be a part of it, [even though] you might be just a small cog in the whole wheel.”
- Mechanical Turk takes its name from a chess-playing machine that was developed as a parlour trick in the Austrian imperial court in the 18th century: a player would sit down and play a game of chess against the “machine”, which concealed a human chess master inside.
- Mechanical Turk is often touted as “artificial artificial intelligence”, or a “human cloud”. The platform has been around for more than a decade, but the types of tasks are changing as computers become smarter. Now, the workers on the human cloud are helping to train computers, which are refining their own artificial intelligence capabilities to become more human-like.
- With the rise of artificial intelligence and specifically machine learning, in which machines teach themselves to recognise patterns by analysing the data they are given, the task of the trainer has become even more important.
- Other machine training projects are being conducted on the human cloud by Google, Twitter, and by Amazon, which owns Mechanical Turk. One of Amazon’s earliest uses of the platform for machine-training was asking humans to check whether its algorithm had correctly identified duplicate retail products on its shopping website.
Panos Ipeirotis, an associate professor at New York University’s Stern School of Business, who studies crowdsourcing, says tasks on Mechanical Turk are changing and adapting as computers become more capable.
“You have a lot of work that is like quality control of the output of computer processes or AI [artificial intelligence] processes,” he explains. “We have more tasks with AI, so we need more humans to verify [the output].”
Compared with when the service first launched in 2005, it is now easier for computers to identify images, read text, and even write sentences. “It used to be humans writing the caption for an image,” says Mr Ipeirotis. “Now computers are writing the caption and humans are checking the caption.”
- Online forums are filled with workers complaining about how difficult it is to communicate with the people assigning the tasks.
- This training function is helped by the fact that many of the workers on Mechanical Turk, which is the largest English-language crowdsourcing platform, are highly educated. A recent study from the Pew Research Center found that one in two Turkers has a college degree, compared with a third of the US workforce overall.
“The biggest surprise was probably the education level of the people doing the tasks,” says Paul Hitlin, senior researcher at Pew, referring to the study, which surveyed more than 3,000 Turkers. “Usually you would expect low-paying jobs to attract less educated workers. But what we found here is that Turkers tend to be more educated than the working public in general.”
Companies using Mechanical Turk for machine training are likely to be paying a fraction of what it would cost to have full-time employees sort and click through pictures.
Wages on the platform, where workers are paid per task rather than per hour, are usually below the US federal minimum wage of $7.25 per hour. The Pew survey found about half of Turkers make less than $5 per hour. Nearly two-thirds of the tasks posted on the site pay 10 cents or less.
Not all the tasks on Mechanical Turk are machine training — chores like transcription represent about quarter of the tasks posted, while identifying information seen in sales receipts is about a fifth of tasks posted, according to Pew. While the tasks vary, they all share one thing: being difficult for computers, and relatively easy for humans.
Among the workers, many enjoy the flexibility of being able to make money when they choose simply by going online. But there has been a backlash against what critics say is a faceless platform that offers little recourse when workers are treated unfairly by taskmasters.
- As noted in a Bloomberg article, IBM Watson wanted to create a chatbot for spectators to use at the 2016 Masters golf tournament. Golf fans used on-site tablets or their own smartphones to ask the bot questions or chat with it. The only problem? IBM couldn’t find enough golf-related training data.
- IBM sent Mighty AI volumes of information obtained from the web related to golf. Using this data, Mighty AI found workers familiar with the game, had them tag information unique to the sport and compose questions and answers based on the material. That data became the basis of IBM Watson’s golf conversational agent.
VentureBeat points out that the idea of a self-learning AI system is still largely the realm of science fiction:
‘While AI systems are becoming more intelligent and “aware,” they’re not quite at the stage of being able to teach themselves — there are things that only humans are able to decipher and detect.’
It’s not just golf where Mighty AI can prove its worth. There are opportunities to “democratize coaching” outside of just sports, such as inindustries like retail, healthcare, and life sciences. Like other ‘sharing economy’ platforms, the startup uses ratings and measurement systems to keep track of quality and incentivize contributions. Bloomberg states:
“Mighty AI has more than 100,000 specialists in 155 countries and it rates how they handle tasks. If the person does well, he or she gets paid more and gets offered more jobs. A poor job will generate feedback and eventually lead to termination if the person doesn’t improve.”
- So now what you need to get your AI system up to speed are trainers rather than developers. But how can you quickly train an AI system? Startup Mighty AI uses an innovative crowdsourced approach. The company (recently renamed from Spare5) gathers information from humans, namely subject matter experts who are willing to answer questions about a certain topic. And they get paid for it. Examples include finding golf aficionados for IBM, people who can describe photos for Getty images, and radiologists or technicians to read tumor scans.
- How does it work? And why do you still need some human input if AI is already in play? The human input, which can be via crowdsourcing or a few individuals is needed to train the AI engine, which uses a technique from AI called machine learning to learn from the human(s). Take AIDR, for example. This experimental solution, which stands for Artificial Intelligence for Disaster Response, uses AI powered by crowdsourcing to automatically identify relevant tweets and text messages in an exploding meadow of digital data. The crowd tags tweets and messages they find relevant and the AI engine learns to recognize the relevance patterns in real-time, allowing AIDR to automatically identify future tweets and messages.
As far as we know, AIDR is the only Big Data solution out there that combines crowdsourcing with real-time machine learning for disaster response. Why do we use crowdsourcing to train the AI engine? Because speed is of the essence in disasters. You need a crowd of Digital Humanitarians to quickly tag as many tweets/messages as possible so that AIDR can learn as fast as possible. Incidentally, once you’ve created an algorithm that accurately detects tweets relaying urgent needs after a Typhoon in the Philippines, you can use that same algorithm again when the next Typhoon hits (no crowd needed).
- For years people have dreamed of a robot personal assistant, and products like Facebook M and Clara Labs are making this a reality. But they don’t automate everything. Instead they have algorithms handle emails and scheduling issues where the intent is clear to them and hand more complicated messages and requests to human being.
- Luckily, good machine learning algorithms can tell the cases where they are likely to do well and likely to struggle. Machine models have no ego, so they’re happy to tell you when their confidence is low. This is why the “human-in-the-loop” design pattern has become very widespread: humans get passed the processes and decisions that a machine can’t confidently make.
- Active learning is a design pattern that combines the first two patterns. The training data collected by the “Human in the Loop” can be fed back into the algorithm to make it better. Algorithms learn like people — novel, complicated situations help them learn much faster. So the examples that the algorithm can’t do that get labeled by a human are the perfect examples to help the algorithm improve.
- Byfar the most common kind of artificial intelligence used in the business world is called supervised machine learning. The “supervised” part is important: it means that an algorithm is learning from training data. Algorithms still don’t learn anywhere near as efficiently as humans, but they can make up for it by processing far, far more data.
The quantity and quality of training data is actually the most important factor for ensuring a machine learning algorithm works well and the best companies take this training data collection process very, very seriously. Many people don’t realize that Google pays for tens of millions of man-hours collecting and labeling data that they feed into their machine learning algorithms.
Collecting training data is a never-ending process. Every time Twitter invents a new word or emoji, machine learning algorithms have no way of understanding it until they see many examples of its usage. Every time a company wants to expand into a new language or even a new market with slightly different patterns, they need to collect a new set of training data or their machine learning algorithms are working under dubious circumstances.
As machine learning becomes more well understood and high quality algorithms become something you can buy off the shelf, training data collection has become the most labor intensive part of launching a new machine learning algorithm.
- CrowdFlower cleans up messy and incomplete data using an online workforce of millions of people. Typical users of CrowdFlower are data scientists who utilize the software to create training data to build models and train machine learning algorithms.
The platform allows users to distribute work to contributors in the U.S. and 153 other countries while maintaining quality and controlling costs. On a continuous basis, these contributors discover work on online job boards and decide what they're going to work on based on how interesting it is, how much work is available and how much the job compensates them. These jobs can include analyzing the sentiment of tweets on a brand or hashtag, scoring relevance for search queries and results of an e-commerce website or moderating user generated content.
Once data is uploaded to the platform, the system automatically allocates the work to contributors and tests them against known answers hidden within the task (what CrowdFlower refers to as a "job" ). The way in which contributors perform on these hidden test questions calibrates how much the system trusts them on an individual level. As long as contributors remain trusted they're allowed to continue working on a given job. If they become untrusted, they're removed from the job and all of their work is disregarded. Multiple contributor judgments are collected and an aggregate answer with an associated confidence score (agreement of the contributors weighted by the trust of each contributor) is provided as a result - effectively returning the "most trusted judgment," for a given unit of data.
- CrowdFlower is a data enrichment, data mining and crowdsourcing company based in San Francisco, United States. The company offers a software as a service platform which allows users to access an online workforce of millions of people to clean, label and enrich data.
- “Contrary to popular belief, we don’t believe AI is going to take everyone’s jobs,” says Leith. “We believe the future is a place where AI augments humans’ skills. In this case we’re using AI to give our brewer superhuman skills, enabling them to test and receive feedback on our beer more quickly than ever before.”
IntelligentX insists its method allows brewers to respond to customers’ changing tastes faster than ever before, and the company’s four beers – Golder, Amber, Pale and Black – have apparently been altered around 11 times since they were first brewed.
- One UK company is looking to change all that, though. IntelligentX’s AI might not actually drink beer, but it is learning how to brew it thanks to machine-learning algorithms.
It works like this: IntelligentX has made four different types of beer. People that drink the beer give their feedback to a bot on Facebook Messenger. The company’s algorithm – called Automated Brewing Intelligence (ABI) – uses a mix of reinforcement learning and Bayesian optimization to tell a human brewer how to push a beer’s recipe in one direction or another.
- A great example of combining superficial intelligence with AI is an application I just downloaded on my iPhone called Waze. If you haven't tried it yet, you really should. Like Google Maps or MapQuest, Waze helps you navigate the streets of your locale. You provide an address and mount your phone in your car, and then Wave gives your real-time navigation directions to your destination. What's different about the app is the Waze community, which is actively involved in feeding you superficial data. For instance, with the help of your local community, Waze tells you where there's an accident, construction that requires a detour, or even a cop hiding out under a bridge. Waze combines this information with real-time analytics to determine your best route. It's amazingly powerful and accurate and puts Google to shame. That's what the wisdom of the crowd can do for you.
- Spare5 was spun out of Seattle-based VC Madrona Venture's labs in 2014, a mobile rival to Amazon's Mechanical Turk program, which finds workers for tasks online. Around the same time, Getty, reached out to get help categorizing images in its collection.
Like Getty, customers increasingly wanted Fives, -- the people that take on the "microtasks" -- to perform brief activities that train AI algorithms. So Spare5 refocused and renamed itself around that idea.
``There's an arms race in training data'' for AI, said Chief Executive Officer Matt Bencke.
- Accenture and Intel see Mighty AI as a way to help their customers deploy artificial intelligence apps and algorithms more quickly, letting them use Accenture services to set them up and Intel software and chips to run them.
"What we like about Mighty AI is that for a lot of our customers the first step is annotating data -- they need that before they can build on top of our chips and software for AI," said Ken Elefant, Intel Capital managing director for software and security. "With Mighty AI all of this annotation will happen at a much faster rate which will help Intel customers deploy much more quickly."
Some companies handle the problem by trying to label data themselves, and others use general crowdsourcing software like Mechanical Turk or CrowdFlower, Bencke said.
Mighty AI has more than 100,000 specialists in 155 countries and it rates how they handle tasks. If the person does well, he or she gets paid more and gets offered more jobs. A poor job will generate feedback and eventually lead to termination if the person doesn't improve.
Mighty AI dubs its product "Training Data as a Service," a riff on cloud product categories like "Infrastructure as a Service" and "Software as a Service."
- The company, which renamed itself to reflect a focus on AI training tasks, is adding three new investors as part of a $14 million funding round: Intel Capital, Google Ventures and Accenture Ventures. It's unveiling partnerships with Intel and Accenture, too.