Even beyond the fully autonomous robots that appeared in the second half of the 20th Century, human beings have long been fascinated by automata. From Da Vinci’s mechanical knight  (designed around 1495) to Vaucanson’s “digesting duck” of 1739, and John Dee’s flying beetle of 1543 (which caused him to be charged with sorcery), the history of robotics stretches well before 1941 when Isaac Asimov first coined the term in one of his short stories.

Robotics combines computer science and mechanical engineering to construct robots that are able to assist humans in various tasks. We’re used to seeing industrial robots in manufacturing, for example, in the construction of cars. And, robots have been, and continue to be, particularly useful in heavy industry to help with processes that could be dangerous for humans resulting in injury or even death. Robotic arms, sometimes known as manipulators, were originally utilised in the handling of radioactive or biohazardous materials which would cause damage to human tissues and organs with exposure.

ABB Robotics is one of the leading multinational companies that deals in service robotics for manufacturing. Robotics applications include welding, heavy lifting, assembly, painting and coating, bonding and sealing, drilling, polishing, and palletising and packaging. These are all heavy-duty tasks that require a variety of end effectors, but robotics technology has progressed considerably and can also be seen in healthcare carrying out sophisticated medical procedures.

With the global pandemic, advances in robotics for surgery have been invaluable, allowing surgeons to remotely control the procedure from a safe distance. Now this technology is being developed further to allow surgeons to log in to operating theatres anywhere in the world, using remote control systems on a tablet or a laptop. They can then carry out surgery with the assistance of medical robotics on site. The technology was created by Nadine Hachach-Haram, who grew up in war-torn Lebanon, and there is no doubt it will be put to good use in locations where there is medical inequality or conflict or both.    

Mobile robots are also fairly common in various industries but we have yet to see them take over domestic chores in a way that perhaps the inventors of Roomba (the autonomous robotic vacuum cleaner) may have hoped they would. And yet, research has shown that people can tell what kind of personality a Neato Botvac has simply by the way that it moves. The study has provided interesting insights into human-robot interaction. Another study in 2017 demonstrated that anthropomorphic robots actually made people feel less lonely. People who work alongside collaborative robots, whether in the military or in manufacturing, also tend to express affection for their “cobots”.

Building on this affection that it seems people can and do develop for robots, Amazon has created Astro, a rolling droid that incorporates AI and is powered by Alexa, Amazon’s voice assistant. Astro also offers a detachable cup holder if your hands are too full to carry your coffee into the next room. Amazon has revolutionised automation with the use of Kiva robots in its warehouses and data science in its online commerce. So, it will be interesting to see if it can succeed where others have failed in making home robots popular.

What’s the difference between artificial intelligence and robotics?

While artificial intelligence refers only to the computer programs that process immense amounts of information in order to “think”, robotics refers to a machine designed to carry out assistive tasks that don’t necessarily require intelligence. Artificial intelligence has given us neural networks built on machine learning, which mimic the neural pathways of the human brain. It seems like an obvious next step would be to integrate these powerful developments with robotic systems to create intelligent robots.

Tesla’s self-driving cars contain neural networks that use autonomy algorithms to support the car’s real-time object detection, motion planning, and decision-making in complicated real-world situations. This level of advanced architectures and programming in the use of robotics is now being proposed with the Tesla Bot, a humanoid robot that Elon Musk says is designed to eliminate dangerous, repetitive, and boring tasks. Andrew Maynard, Associate Dean at the College of Global Futures, Arizona State University voices caution with regard to “a future that, judging by Musk’s various endeavours, will be built on a set of underlying interconnected technologies that include sensors, actuators, energy and data infrastructures, systems integration and substantial advances in computer power.” He adds that “before technology can become superhuman, it first needs to be human – or at least be designed to thrive in a human-designed world.”

It’s an interesting perspective, and one that goes beyond the usual moral and ethical concerns of whether robots could be used for ill, a theme often explored in science fiction films like The Terminator, Chappie, and Robot and Frank. Science fiction has always provided inspiration for actual technologies but it also serves to reflect back some of our own moral conundrums. If, as Elon Musk has indicated, the Tesla Bot “has the potential to be a generalised substitute for human labour over time,” what does this mean for artificially intelligent robotics? Would we essentially be enslaving robots? And, of course, this is built on the belief that the foundation of the economy will continue to be labour.

Tesla Bot hasn’t yet reached prototype stage yet, so whether Musk’s vision becomes a reality remains to be seen. The kinematics required to support bi-ped humanoid robots, though, are extremely complex. When it comes to bipedal robot design, LEONARDO (LEgs ONboARD drOne) is a quadcopter with legs. Only 76cm tall with an extremely light structure, even with an exoskeleton, LEONARDO looks far from the humanoids that the robotics industry may believe will become embedded in our everyday lives. Yet his multimodal locomotion system solves (or perhaps simply avoids) some of the issues real-world bipedal robots experience related to weight and centre of gravity. Mechatronics is an interdisciplinary branch of engineering that concerns itself with these kinds of issues. Some would say that mechatronics is where control systems, computers, electronic systems, and mechanical systems overlap. However, others would say that it’s simply a buzzword which is interchangeable with automation, robotics, and electromechanical engineering.

Build your knowledge in artificial intelligence with an MSc

Artificial intelligence is vital to take robotics research to the next level and enable robots to go beyond the relatively simple tasks they can complete on their own, as well as the more complex tasks they support humans with. Research in areas such as machine learning, distributed artificial intelligence, computer vision, and human-machine interaction will all be key to the future of robotics. 

Inspired to discover more about how you can specialise in AI? Study an 100% online and part-time MSc Computer Science with Artificial Intelligence with the University of York.

Reinforcement learning (RL) is a subset of machine learning that allows an AI-driven system (sometimes referred to as an agent) to learn through trial and error using feedback from its actions. This feedback is either negative or positive, signalled as punishment or reward with, of course, the aim of maximising the reward function. RL learns from its mistakes and offers artificial intelligence that mimics natural intelligence as closely as it is currently possible.

In terms of learning methods, RL is similar to supervised learning only in that it uses mapping between input and output, but that is the only thing they have in common. Whereas in supervised learning, the feedback contains the correct set of actions for the agent to follow. In RL there is no such answer key. The agent decides what to do itself to perform the task correctly. Compared with unsupervised learning, RL has different goals. The goal of unsupervised learning is to find similarities or differences between data points. RL’s goal is to find the most suitable action model to maximise total cumulative reward for the RL agent. With no training dataset, the RL problem is solved by the agent’s own actions with input from the environment.

RL methods like Monte Carlo, state–action–reward–state–action (SARSA), and Q-learning offer a more dynamic approach than traditional machine learning, and so are breaking new ground in the field.

There are three types of RL implementations: 

• Policy-based RL uses a policy or deterministic strategy that maximises cumulative reward
• Value-based RL tries to maximise an arbitrary value function
• Model-based RL creates a virtual model for a certain environment and the agent learns to perform within those constraints

How does RL work?

Describing fully how reinforcement learning works in one article is no easy task. To get a good grounding in the subject, the book Reinforcement Learning: An Introduction by Andrew Barto and Richard S. Sutton is a good resource.

The best way to understand reinforcement learning is through video games, which follow a reward and punishment mechanism. Because of this, classic Atari games have been used as a test bed for reinforcement learning algorithms. In a game, you play a character who is the agent that exists within a particular environment. The scenarios they encounter are analogous to a state. Your character or agent reacts by performing an action, which takes them from one state to a new state. After this transition, they may receive a reward or punishment. The policy is the strategy which dictates the actions the agent takes as a function of the agent’s state as well as the environment.

To build an optimal policy, the RL agent is faced with the dilemma of whether to explore new states at the same time as maximising its reward. This is known as Exploration versus Exploitation trade-off. The aim is not to look for immediate reward, but to optimise for maximum cumulative reward over the length of training. Time is also important – the reward agent doesn’t just rely on the current state, but on the entire history of states. Policy iteration is an algorithm that helps find the optimal policy for given states and actions.

The environment in a reinforcement learning algorithm is commonly expressed as a Markov decision process (MDP), and almost all RL problems are formalised using MDPs. SARSA is an algorithm for learning a Markov decision. It’s a slight variation of the popular Q-learning algorithm. SARSA and Q-learning are the two most typically used RL algorithms.

Some other frequently used methods include Actor-Critic, which is a Temporal Difference version of Policy Gradient methods. It’s similar to an algorithm called REINFORCE with baseline. The Bellman equation is one of the central elements of many reinforcement learning algorithms. It usually refers to the dynamic programming equation associated with discrete-time optimisation problems.

The Asynchrous Advantage Actor Critic (A3C) algorithm is one of the newest developed in the field of deep reinforcement learning algorithms. Unlike other popular deep RL algorithms like Deep Q-Learning (DQN) which uses a single agent and a single environment, A3C uses multiple agents with their own network parameters and a copy of the environment. The agents interact with their environments asynchronously, learning with every interaction, contributing to the total knowledge of a global network. The global network also allows agents to have more diversified training data. This mimics the real-life environment in which humans gain knowledge from the experiences of others, allowing the entire global network to benefit.

Does RL need data?

In RL, the data is accumulated from machine learning systems that use a trial-and-error method. Data is not part of the input that you would find in supervised or unsupervised machine learning.

Temporal difference (TD) learning is a class of model-free RL methods that learn via bootstrapping from a current estimate of the value function. The name “temporal difference” comes from the fact that it uses changes – or differences – in predictions over successive time steps to push the learning process forward. At any given time step, the prediction is updated, bringing it closer to the prediction of the same quantity at the next time step. Often used to predict the total amount of future reward, TD learning is a combination of Monte Carlo ideas and Dynamic Programming. However, whereas learning takes place at the end of any Monte Carlo method, learning takes place after each interaction in TD.

TD Gammon is a computer backgammon program that was developed in 1992 by Gerald Tesauro at IBM’s Thomas J. Watson Research Center. It used RL and, specifically, a non-linear form of the TD algorithm to train computers to play backgammon to the level of grandmasters. It was an instrumental step in teaching machines how to play complex games.

Monte Carlo methods represent a broad class of algorithms that rely on repeated random sampling in order to gain numerical results that point to probability. Monte Carlo methods can be used to calculate the probability of:

• an opponent’s move in a game like chess
• a weather event occurring in the future
• the chances of a car crash under specific conditions

Named after the casino in the city of the same name in Monaco, Monte Carlo methods first arose within the field of particle physics and contributed to the development of the first computers. Monte Carlo simulations allow people to account for risk in quantitative analysis and decision making. It’s a technique used in a wide variety of fields including finance, project management, manufacturing, engineering, research and development, insurance, transportation, and the environment.

In machine learning or robotics, Monte Carlo methods provide a basis for estimating the likelihood of outcomes in artificial intelligence problems using simulation. The bootstrap method is built upon Monte Carlo methods, and is a resampling technique for estimating a quantity, such as the accuracy of a model on a limited dataset.

Applications of RL

RL is the method used by DeepMind to initiate artificial intelligence in how to play complex games like chess, Go, and shogi (Japanese chess). It was used in the building of AlphaGo, the first computer program to beat a professional human Go player. From this grew the deep neural network agent AlphaZero, which taught itself to play chess well enough to beat the chess engine Stockfish in just four hours.

AlphaZero has only two parts: a neural network, and an algorithm called Monte Carlo Tree Search. Compare this with the brute force computing power of Deep Blue, which, even in 1997 when it beat world chess champion Garry Kasparov, allowed the consideration of 200 million possible chess positions per second. The representations of deep neural networks like those used by AlphaZero, however, are opaque, so our understanding of their decisions is restricted. The paper Acquisition of Chess Knowledge in AlphaZero explores this conundrum.

Deep RL is being proposed in the use of unmanned spacecraft to navigate new environments, whether it’s Mars or the Moon. MarsExplorer is an OpenAI Gym compatible environment that has been developed by a group of Greek scientists. There are four deep reinforcement learning algorithms that the team has trained on the MarsExplorer environment, A3C, Ranbow, PPO, and SAC, with PPO performing best. MarsExplorer is the first open-AI compatible reinforcement learning framework that is optimised for the exploration of unknown terrain.

Reinforcement learning is also used in self-driving cars, in trading and finance to predict stock prices, and in healthcare for diagnosing rare diseases.

Deepen your learning with a Masters

These complex learning systems created by reinforcement learning are just one facet of the fascinating and ever-expanding world of artificial intelligence. Studying a Masters degree can allow you to contribute to this field, which offers numerous possibilities and solutions to societal problems and the challenges of the future. 

The University of York offers a 100% online MSc Computer Science with Artificial Intelligence to expand your learning, and your career progression.

What is deep learning?

Deep learning is a sector of artificial intelligence (AI) concerned with creating computer structures that mimic the highly complex neural networks of the human brain. Because of this, it is also sometimes referred to as deep neural learning or deep neural networks (DNNs). 

A subset of machine learning, the artificial neural networks utilised in deep learning are capable of sorting much more information from large data sets to learn and consequently use in making decisions. These vast amounts of information that DNNs scour for patterns are sometimes referred to as big data.

Is deep learning machine learning?

The technology used in deep learning means that computers are closer to thinking for themselves without support or input from humans (and all the associated benefits and potential dangers of this). 

Traditional machine learning requires rules-based programming and a lot of raw data preprocessing by data scientists and analysts. This is prone to human bias and is limited by what we are able to observe and mentally compute ourselves before handing over the data to the machine. Supervised learning, unsupervised learning, and semi-supervised learning are all ways that computers become familiar with data and learn what to do with it. 

Artificial neural networks (sometimes called neural nets for short) use layer upon layer of neurons so that they can process a large amount of data quickly. As a result, they have the “brain power” to start noticing other patterns and create their own algorithms based on what they are “seeing”. This is unsupervised learning and leads to technological advances that would take humans a lot longer to achieve. Generative modelling is an example of unsupervised learning.

Real-world examples of deep learning

Deep learning applications are used (and built upon) every time you do a Google search. They are also used in more complicated scenarios like in self-driving cars and in cancer diagnosis. In these scenarios, the machine is almost always looking for irregularities. The decisions the machine makes are based on probability in order to predict the most likely outcome. Obviously, in the case of automated driving or medical testing, accuracy is more crucial, so computers are rigorously tested on training data and learning techniques.

Everyday examples of deep learning are augmented by computer vision for object recognition and natural language processing for things like voice activation. Speech recognition is a function that we are familiar with through use of voice-activated assistants like Siri or Alexa, but a machine’s ability to recognise natural language can help in surprising ways. Replika, also referred to as “My AI Friend”, is essentially a chatbot that gets to know a user through questioning. It uses a neural network to have an ongoing one-to-one conversation with the user to gather information. Over time, Replika begins to speak like the user, giving the impression of emotion and empathy. In April 2020, at the height of the pandemic, half a million people downloaded Replika, suggesting curiosity about AI but also a need for AI, even if it does simply mirror back human traits. This is not a new idea as in 1966, computer scientist Joseph Weizenbaum created what was a precursor to the chatbot with the program ELIZA, the computer therapist.

How does deep learning work?

Deep learning algorithms make use of very large datasets of labelled data such as images, text, audio, and video in order to build knowledge. In its computation of the content – scanning through and becoming familiar with it – the machine begins to recognise and know what to look for. Like the human brain, each computer neuron has a role in processing data, it provides an output by applying the algorithm to the input data provided. Hidden layers contain groups of neurons.

At the heart of machine learning algorithms is automated optimisation. The goal is to achieve the most accurate output so we need the speed of machines to efficiently assess all the information they have and to begin detecting patterns which we may have missed. This is also core to deep learning and how artificial neural networks are trained.

TensorFlow is an open source platform created by Google, written in Python. A symbolic maths library, it can be utilised for many tasks, but primarily for training, transfer learning, and developing deep neural networks with many layers. It’s particularly useful for reinforcement learning because it can calculate large numbers of gradients. The gradient is how the data is seen on a graph. So, for example, the gradient descent algorithm would be used to minimise error function and would be represented graphically as the gradient at its lowest possible point. The algorithm used to calculate the gradient of an error function is “backpropagation”, short for “backward propagation of errors”.

One of the most used deep learning models in reinforcement learning, particularly for image recognition, Convolutional Neural Networks (CNN) can learn increasingly abstract features by using deeper layers. CNNs can be accelerated by using Graphics Processing Units (GPUs) because they can process many pieces of data simultaneously. They can help perform feature extraction by analysing pixel colour and brightness or vectors in the case of grayscale.

Recurrent Neural Networks (RNNs) are considered state of the art because they are the first of their kind to use an algorithm that lets them remember their input. Because of this, RNNs are used in speech recognition and natural language processing in applications like Google Translate.

Can deep learning be used for regression?

Neural networks can be used for both classification and regression. However, regression models only really work well if they’re the right fit for the data and that can affect the network architecture. Classifiers in something like image recognition, have more of a compositional nature compared with the many variables that can make up a regression problem. Regression offers a lot more insight than simply, “Can we predict Y given X?”, because it explores the relationship between variables. Most regression models don’t fit the data perfectly, but neural networks are flexible enough to be able to pick the best type of regression. To add to this, hidden layers can always be added to improve prediction.

Knowing when to use regression or not to solve a problem may take some research. Luckily, there are lots of tutorials online to help, such as How to Fit Regression Data with CNN Model in Python.

Ready to discover more about deep learning?

The University of York’s online MSc Computer Science with Artificial Intelligence from the University of York is the ideal next step if your career ambitions lie in this exciting and fast-paced sector. 

Whether you already have knowledge of machine learning algorithms or want to immerse yourself in deep learning methods, this master’s degree will equip you with the knowledge you need to get ahead.