Showcasing how Python is used in various fields with use cases
Python is one of the most dominant programming languages used today and as such is used within a variety of professions to solve a range of problems.
If you are in one of the industries or fields discussed in this article it is very likely that Python will have an influence within your line of work, even if you do not interact with Python code directly.
It is therefore a very reasonable endeavour to break down where and how exactly Python is being used and understand why – this is what this article will effectively do for the reader.
Why is Python so Popular?
Before we jump into the various ways Python is used in industry, let’s firstly analyse why Python is indeed so popular right now.
Read up on all the in-demand programming languages of 2021 from our article dedicated to the subject: The Best Programming Languages to Learn.
Survey Consensus: Developers Love Python
Python is the 3rd most loved language and ranking 4 as the most popular programming language period according to the Stack Overflow Developer Survey 2020. As a Python programmer you can expect an average salary of $120k in the US, or $459k globally – a healthy amount that is in the upper bracket of programming salaries – there is indeed a healthy demand for Python developers.
Other developer surveys corroborate these findings. The Coding Game Developer Survey 2020 convincingly places Python as one of the most known programming languages, alongside C++ and JavaScript, and actually ranks Python as the most loved language with their findings.
Develop Intelligence’s 2020 results also show that Python is the fastest growing language . This is partly due to its adoption in rapidly emerging fields in artificial intelligence (AI) and data science.
What Makes the Python Programming Language Popular
Python is a general purpose programming language, which by definition means that Python programs can be applied to a range of software problems in a whole assortment of domains, ranging from AI and Machine Learning to web development, game development and FinTech.
These (and more) are all industries that we will be discussing further down.
This generality is very good news for the Python programmer interested in migrating to another space in the industry, whereby learned Python skills from one domain can be applied to others quite naturally.
This further amplifies Python’s attraction as a core programming language to learn.
Along with Python’s trait of being a general purpose programming language, these other characteristics have aided in its rise to dominance.
Easy to learn with expressive syntax
Python is relatively easy to learn with its simple, expressive syntax, memory management under the hood, and emphasis on conventions and readability.
This ease of use is partially what separates Python from other general purpose programming languages such as C++, C#, Java and Go – all of which have steeper learning curves.
Open source with multi platform support
Python is open source and has a large developer community dedicated to maintaining and upgrading the language specs. With the first implementation constructed in 1989, Python has stood the rest of time and is now on version 3.9.4 (at the time of writing), released on the 4th April 2021. Python 3 is the dominant version of the programming language.
Python also runs on all major operating systems, from the server to the home desktop, laptop or Mac (even M1 macs from version 3.9.1).
A vibrant ecosystem of packages
A wealth of Python libraries and utilities already exist in extensively-used package managers, namely PyPI and Conda. The former being a general repository for Python packages and the latter being more focused on the data scientist community.
Regularly maintained and updated
To support the regular updates the language receives, tools like Virtual Environment ensure that developers can have dedicated Python workspaces, each with a particular Python version installed.
Without getting too technical here, Python is ultimately a very well supported, maintained and adopted project.
There’s a library for everything. We’ve been learning a lot of these technologies and a lot of these architectures as we go. And, so, when I don’t understand connection pools, I can just find a library until I understand it better myself and write our own. Don’t understand web frameworks, so we’ll use someone else’s until we make our own… Python has an awesome crutch like that.
Python Usage by Industry
Starting with the fastest growing space, let’s explore how Python is used in data science and Machine Learning.
Python for Data Science
Data science is Python’s primary use case today, having an abundance of highly efficient packages and tools for statistics, analytics problems, data visualisation, as well as the fast growing domain of artificial intelligence and deep learning.
The ecosystem of Machine Learning in Python has become so vast that the next section will be dedicated to that specifically.
The core packages for data science
At the core of Python’s data science offerings is NumPy, billed as a fundamental package for scientific computing in Python.
NumPy has a wealth of formulas and advanced data structures like matrices built-in to the package that are highly optimised and efficient to implement.
NumPy has paved the way for other packages in the ecosystem to enhance its offering, including:
- Matplotlib: a comprehensive library of data visualisation utilities that allow developers to visualise their data.
- Scikit Learn: A Machine Learning focused library for predictive analysis.
- Pandas: A fast data analysis and manipulation tool that facilitates working with a range of data types and formats.
- SciPy: Bundles NumPy, Matplotlib and Pandas, as well as other tools to create an ecosystem of tools for mathematics, science and engineering.
These foundational Python packages are paramount in data science today, and are used in both industry and academia by engineers, scientists and researchers.
Notable case studies
Looking into NumPy’s case studies will reveal the importance and overall reliance of the package. The first image of a blackhole recorded by the Event Horizon Telescope (EHT) was reconstructed and evaluated using NumPy and the ecosystem of tools around it. NumPy helped in solving complex maths and data manipulation at high speed when analysing gravitational waves picked up by the Laser Interferometer Gravitational-Wave Observatory (LIGO).
These are of course highly advanced cutting edge research projects, but they show the magnitude of Python’s integration in both industry and academia. It is already a fair assumption that all major technology companies are using Python within their programming stack. This will become more apparent as we progress through more industries.
This core ecosystem of scientific computing made Python well-poised for the fast growing and increasingly important Machine Learning space, and the field of artificial intelligence as a whole.
Python for AI and Machine Learning
The inevitable rise of Machine Learning
Python has now achieved the very impressive feat of becoming the de facto programming language for Machine Learning (ML) applications by usage, being the home of the major Machine Learning frameworks big players rely upon today.
The field of Artificial intelligence is now exploding in popularity as deep learning applications become more sophisticated; this has been fuelled by the progression of key technologies, including:
- The advent of big data and the ability to host large amounts of data in the cloud.
- Advancements in GPU technology, allowing AI applications – and neural networks in-particularly – to grow in complexity and capability.
- Smartphones and camera / microphone accessibility that everyone in developed nations have access to. This has led to an abundance of data that Machine Learning applications require to be trained.
What is Python used for in Machine Learning
Machine Learning frameworks are what really kicked off Python’s adoption in the space, predominantly by virtue of Google’s TensorFlow framework and PyTorch, another framework conceived by engineers at Facebook. Beyond these flagship frameworks are products like Keras that further simplify the learning process of newcomers to ML.
TensorFlow is geared towards production use cases whereas PyTorch is more for experimental or research based projects with its easier-to-use APIs. Nevertheless, large chunks of both projects are written in C++ and are highly optimised for crunching numbers.
Every company with an interest in deep learning will have either adopted TensorFlow or PyTorch. TensorFlow has gained adoption from huge names including AirBnB, Coca Cola, Google’s Deep Mind, Intel, Twitter, and a lot more. PyTorch has had similar success with names like Facebook, Walmart, Amazon, Northrop Grumman, and Verizon Wireless.
Machine learning is now applied to our every-day life, from processing search requests to organising photo albums, communicating with a virtual assistant, authenticating your fingerprint, and optimising photos – and these are all on your smartphone. Needless to say, Python will be cemented in the ML ecosystem for the foreseeable future.
To read up on Machine Learning programming languages and frameworks of today, refer to our published piece dedicated to the topic: Machine Learning Programming Languages and Frameworks for 2021.
Python has been an important part of Google since the beginning, and remains so as the system grows and evolves. Today dozens of Google engineers use Python, and we’re looking for more people with skills in this language.
Python for Robotics and Reinforcement Learning
Another field that has quickly emerged into an active area of research is Reinforcement Learning (RL). Reinforcement learning takes the approach of learning a particular action given a state of an environment. The robot, or agent, constantly evaluates its environment and attempts to choose an optimal action for any given time step.
This seemingly simple concept quickly becomes a complex mesh of algorithms, advanced maths and is somewhat experimental in application. Similarly to Machine Learning, Python provides the foundational platform on which Reinforcement research and application development can take place by utilising some of the packages we have already discussed, along with some new ones.
OpenAI and Gym ecosystem
OpenAI being one of the leaders in Reinforcement Learning research has open sourced a lot of groundbreaking software since its inception in 2015, notably their Gym toolkit for studying and developing Reinforcement Learning algorithms.
RL models have been trained on popular games such as Space Invaders that are freely available to download and tinker with for the researcher.
But even with Python, RL still has a steep learning curve and a high barrier to entry. Oftentimes agents are trained in a simulated space, and therefore rely on physics simulators such as Mujoco or Box2D.
Developing RL is more than learning algorithms and implementing them in Python; Python also needs to run and manage the simulated environment, handle Physics and plot the resulting data.
Reinforcement Learning in the commercial sector
So who is undertaking this challenging field of research? Boston Dynamics have demonstrated the capabilities of Reinforcement learning with their real-world robots, the most notable of which being Spot., a 4-legged robot capable of walking, running, climbing stairs, self charging, and picking itself up when it falls over.
Tesla’s autopilot software relies on RL for training the vehicles amongst other Machine Learning techniques in visual computing. Self driving efforts will undoubtedly rely on Python toolkits for their development.
Python for Web Development
Python has strong adoption in web development. Given its multi-platform support, ease of use, open source nature and generality, Python is well suited for the server and web applications in general.
Server Applications
Although Python has some stiff competition in the web development space from JavaScript and Node.js runtime, it still offers some elegant tools for web applications and web infrastructure in general.
The flagship Python web framework is Django which is currently being used by companies like Instagram, Mozilla, Pinterest, Open Knowledge Foundation and OpenStack.
Another light-weight web server Flask is also a popular choice for Python developers to host websites. It is well suited for microservice applications but also does a great job of hosting static websites such as personal blogs or portfolio web pages.
There are a few offerings with a similar nature to Flask, such as Pyramid, Bottle, and Web2Py that all focus on similar traits; they’re fast, lightweight and simple frameworks.
Data and Image Processing
Beyond frameworks, Python files can also be run in a standalone manner on the server and carry out specific tasks:
File management
File management is a simple task that is supplied out of the box from Python’s standard library (found here) with modules such as os, pathlib, and glob, dealing with operating systems tasks pertaining to file management, file system paths and pathname pattern expansion respectively.
Image manipulation
Python libraries such as scikit-image contain a collection of standard algorithms for image processing, with OpenCV being another option that focuses more on Machine Learning computer vision techniques.
Database communication and data processing
Python supports SQL out of the box with sqllite3 and major databases have reliable Python drivers for database communication. The MySQL connector connects Python to any external MySQL database, Psycopg is the most popular PostgreSQL adapter, and PyMongo is for MongoDB, and the list goes on.
Data Mining and Web Scraping
Web scraping is the process of crawling the web and extracting data from the web pages visited. One such pipeline could entail Python’s requests module for connecting to external websites, before extracting the HTML with Beautiful Soup (a package designed specifically for HTML data extraction).
In a world where data has become a valuable asset, analytics firms will particularly be interested in such activities and rely on Python’s battle tested pipeline to do so.
We initially chose to use Python because of its reputation for simplicity and practicality, which aligns well with our philosophy of ‘do the simple thing first.
Python in Finance and FinTech
Speaking of analytics and data mining, another industry where data processing tasks are frequently carried out is in finance. Concretely, Python is well suited for FinTech companies because of the range of APIs and tools that support common tasks in finance.
API feeds
Fetching data from real-time price feeds of various stocks, commodities and other assets.
Scraping data
Scraping data from data-rich platforms to analyse the state of a particular market, such as house pricing in a targeted area or state.
Spreadsheet management
Generating and aggregating spreadsheets in an automated manner. Excel (and more recently Google sheets) are heavily relied upon in finance, but handling large amounts of files is cumbersome and time consuming.
What Python can do rather effortlessly is aggregate data, format that data in a spreadsheet (such as a generic CSV file) and output these spreadsheets (either appending to an existing spreadsheet, or creating a new one) in the desired format.
Data management automation
Accelerating data management is what gives finance firms leverage, and Python is used to do this. Imagine organising spreadsheets by date and having a separate folder for storing spreadsheets for each day.
This would be a terribly tedious task for a human, but Python can do this effortlessly and automatically. Repetitive tasks for data collection, aggregation, and output are dramatically speeded up with Python.
Python is an ideal programming language to carry out such tasks due to the same traits we’ve previously mentioned; easy to read, high level syntax, maintainability, multi-platform support, and the huge library of packages readily available.
Jupyter Notebooks for data analytics
Jupyter Notebooks play a key role in Finance with the ability to present data in a dynamic way; dynamic meaning that Python code can be run within the notebook that can generate up-to-date data and report real-time findings when the notebook is run.
Jupyter Notebooks are also widely used in data science fields we discussed above and are not limited to just Python code, although the majority of usage is with Python. They are embedded in learning platforms and often host tutorials and walkthroughs for students.
There is overlap here with many tools we discussed in the Machine Learning section, with pandas, NumPy, and SciPy all being useful tools for data manipulation and visualisation.
Automated trading and bots
Python is also the home of trading bots, that are widely used by FinTech companies as well as hobbyists. Popular packages like FinmarketPy allow you to analyse market data and to backtest trading strategies using their simple APIs. Pyalgotrade is another package used for algorithmic trading that supports both paper-trading and live-trading.
Both traditional assets and Cryptocurrencies are automatically traded with these platforms, with the latter booming in popularity and interest in the finance space as digital assets become more accepted by the mainstream.
Python in Education
Python is ideal as a teaching programming language and as such is widely used in education and online learning, both at an entry level for newcomers and at an advanced level for research students. This is again due to Python’s strengths as a general purpose programming language that we will briefly break down in more detail.
Code readability
Python syntax resembles the English language more-so than other object oriented programming languages (C# and Java are the main competitors of Python). This intentional characteristic of Python positions it as a great entry-level programming language for students endeavouring into programming paradigms for the first time.
Extensibility and reusability
Reusability is critical in any programming task because programmers can rely on battle-tested packages that are widely used and maintained. With one command these packages can be installed and imported into any project. Python makes no compromises here.
Cross platform run-ability
Whether education institutions opt for Windows machines, Macs or linux based environments, Python is well supported and compatible with all major operating systems. You can even run Python in the cloud with services like Python Anywhere.
This is important in educational settings where there is no guarantee which platforms each institution will adopt.
Python also runs on the Raspberry Pi and even microcontrollers – small, low-powered chips designed to run within embedded devices and appliances (think the internet of things, or IoT). These cheap pieces of hardware make for great learning platforms for students and enthusiasts.
Python for Video Game Development
Although Python is not an optimal video game development solution by today’s standards and tools available, it has nonetheless attracted a community of game developers, and some interesting video game libraries written in Python exist today, thus cementing its place among learning platforms for game development.
Pygame
Pygame 2 is a free and open source cross-platform library for the software development of multimedia applications like video games using Python. A large chunk of the codebase for Pygame is written in C, and games will be very performant as a result of this. It is a capable library with a wealth of projects to browse on their website.
Pyglet
Pyglet is another library boasting similar characteristics to Pygame, Pyglet is another actively maintained library for game development that is purely written in Python. One of Pyglet’s flagship features is support for platform-native windows, allowing you to take advantage of multiple windows and multi-monitor desktops.
These packages are very capable of prototyping ideas, learning how to manage multimedia assets, or just simply learning more of Python and its capabilities in an education setting.
Python in Specialised Fields
We’ve now covered where Python is used most, but what is interesting to know is that Python can also be applied in some very specialised fields due to its generality.
The content in this section covers some interesting ways the language is being applied to solve cutting-edge problems.
Ethical Hacking
Python comes with a wealth of packages that white-hat hacking communities can use to discover vulnerabilities in software.
Discovering vulnerabilities in software and services requires expertise in fields like networking, cryptography and low level programming.
Learning platforms are increasingly seeing the value of such activities in the so-called “white hat” hacking community; or hacking to improve software integrity and security.
Such discoveries come with huge bounties these days, with tech giants like Apple having their own bug bounty programs.
The following are Python packages used for ethical hacking:
- Scrapy: An interactive packet manipulation program, able to forge or decode packets of a range of protocols, send them on the wire, capture them, match requests and replies, and much more.
- IMPacket: A collection of Python classes for working with network protocols.
- Cryptography: A package which provides cryptographic recipes and primitives to Python developers.
- Python-Nmap: A library which helps in using nmap port scanner. It allows you to easily manipulate nmap scan results.
- Pylibnet: Python module for the libnet packet injection library.
Astronomy
Python has an active community in astronomy and space exploration who have developed interesting data visualisation applications:
- Astropy: A community effort to develop a core package for astronomy using Python and improve usability, interoperability, and collaboration between astronomy Python packages. Example snippets can be browsed on their gallery page.
- SunPy: A package billed as a community-developed, free and open-source solar data analysis environment for Python.
- SpacePy: a package targeted at the space sciences, that aims to make basic data analysis, modelling and visualization easier.
Pharmacology and Medicine
Python’s wealth of data visualisation libraries have given rise to pharmacology based applications, such as analysing medical data, clinical diagnoses based on the patients’ medical records and symptoms, and making computational models to speed up the process of development of new medications.
Cutting-edge deep learning techniques are also actively being researched to aid in these tasks.
PyGeno is a package aiming to make extracting data such as gene sequences a breeze, and is designed to be able cope with huge queries. MedPy is a library and script collection for medical image processing in Python.
AstraZeneca success story
As a Python success story, Researchers at AstraZeneca used models formulated in Python to simulate the chemical structure of molecules and their effect in the body. This helped scientists identify potential molecules for new drugs and start testing them more quickly in the laboratory.
Summary
This article has covered where the majority of Python activity is taking place by various industries and their use cases pertaining to software development.
Due to Python’s versatility as a general purpose programming language, it can be applied to a multitude of challenging problems where software created with the language can be applied to discover complex cutting edge solutions.
Because of this, Python is heavily used in research efforts in a variety of fields that we briefly touched on in the specialised fields section. Data science has fast become the flagship use case of Python.
As a very widely adopted and in-demand language that is embedded in a wealth of industries and large corporations, Python is one of, if not the most relevant language of today.
