The Ultimate Python Handbook: From Fundamentals to Advanced Engineering
A deep dive into the entire Python ecosystem. Whether you are initializing your first variable, architecting complex Object-Oriented systems, or optimizing concurrency with asyncio, this guide bridges the gap between basic scripting and professional software engineering standards.
Fundamental
Installation and Environment Setup
Install Python from python.org or use pyenv (recommended at Google) to manage multiple Python versions; on macOS use brew install pyenv, on Linux use the installer script, and always add Python to your PATH. Verify with python --version.
# Recommended: pyenv installation curl https://pyenv.run | bash pyenv install 3.12.0 pyenv global 3.12.0
Python Interpreter and REPL
The REPL (Read-Eval-Print-Loop) is an interactive shell where you can execute Python code line-by-line, perfect for testing snippets and exploring APIs; launch it by typing python or python3 in your terminal.
$ python3
>>> 2 + 2
4
>>> print("Hello")
Hello
>>> exit()
Variables and Data Types
Python is dynamically typed—variables are references to objects and don't require type declarations; core types include int, float, str, bool, and NoneType.
name = "Alice" # str age = 30 # int height = 5.9 # float is_active = True # bool data = None # NoneType # Check type print(type(name)) # <class 'str'>
Basic Operators
┌─────────────────────────────────────────────────────────┐
│ OPERATOR TYPES │
├──────────────┬──────────────────────────────────────────┤
│ Arithmetic │ + - * / // % ** │
│ Comparison │ == != < > <= >= │
│ Logical │ and or not │
│ Assignment │ = += -= *= /= //= **= │
│ Identity │ is is not │
│ Membership │ in not in │
└──────────────┴──────────────────────────────────────────┘
x = 10 // 3 # 3 (floor division) y = 10 ** 2 # 100 (exponent) z = 10 % 3 # 1 (modulo)
String Operations and Formatting
Strings are immutable sequences supporting slicing, concatenation, and multiple formatting approaches; prefer f-strings (Python 3.6+) for readability and performance.
name, age = "Alice", 30 # F-string (preferred) msg = f"Name: {name}, Age: {age}" # Other methods msg = "Name: {}, Age: {}".format(name, age) msg = "Name: %s, Age: %d" % (name, age) # Operations s = "hello" s.upper() # "HELLO" s[1:4] # "ell" s.split('e') # ['h', 'llo'] "lo" in s # True
Lists, Tuples, Sets, Dictionaries
┌────────────┬─────────┬───────────┬───────────┬──────────────┐
│ Type │ Syntax │ Ordered │ Mutable │ Duplicates │
├────────────┼─────────┼───────────┼───────────┼──────────────┤
│ List │ [1,2,3] │ ✓ │ ✓ │ ✓ │
│ Tuple │ (1,2,3) │ ✓ │ ✗ │ ✓ │
│ Set │ {1,2,3} │ ✗ │ ✓ │ ✗ │
│ Dict │ {k: v} │ ✓ (3.7+) │ ✓ │ keys: ✗ │
└────────────┴─────────┴───────────┴───────────┴──────────────┘
lst = [1, 2, 3] # List tup = (1, 2, 3) # Tuple (immutable) st = {1, 2, 3} # Set (unique values) dct = {"a": 1, "b": 2} # Dictionary
Control Flow (if/elif/else)
Python uses indentation (4 spaces per PEP 8) to define code blocks instead of braces; conditions don't require parentheses.
score = 85 if score >= 90: grade = "A" elif score >= 80: grade = "B" elif score >= 70: grade = "C" else: grade = "F" # Ternary operator result = "pass" if score >= 60 else "fail"
Loops (for, while)
Use for to iterate over sequences and while for condition-based loops; break exits the loop, continue skips to next iteration, and else runs if no break occurred.
# For loop with range for i in range(5): # 0, 1, 2, 3, 4 print(i) # Iterating with index for idx, val in enumerate(["a", "b", "c"]): print(idx, val) # While loop count = 0 while count < 5: count += 1 else: print("Completed!") # Runs if no break
List Comprehensions
A concise, Pythonic way to create lists by combining a loop and optional condition into a single expression; faster than equivalent for-loops.
# Basic: [expression for item in iterable] squares = [x**2 for x in range(5)] # [0, 1, 4, 9, 16] # With condition evens = [x for x in range(10) if x % 2 == 0] # [0, 2, 4, 6, 8] # Nested matrix = [[i*j for j in range(3)] for i in range(3)] # Dict/Set comprehensions sq_dict = {x: x**2 for x in range(5)} sq_set = {x**2 for x in range(5)}
Functions and Parameters
Functions are first-class objects defined with def; parameters can have default values, and Python supports positional, keyword, and mixed argument passing.
def greet(name, greeting="Hello", punctuation="!"): return f"{greeting}, {name}{punctuation}" greet("Alice") # "Hello, Alice!" greet("Bob", greeting="Hi") # "Hi, Bob!" greet("Eve", "Hey", "?") # "Hey, Eve?"
Return Values
Functions return None by default; use return to send back values, and Python supports returning multiple values as tuples.
def divide(a, b): if b == 0: return None # Early return quotient = a // b remainder = a % b return quotient, remainder # Returns tuple q, r = divide(10, 3) # Tuple unpacking: q=3, r=1
Scope (Local, Global)
Python follows the LEGB rule for variable resolution: Local → Enclosing → Global → Built-in; use global or nonlocal keywords to modify outer scope variables.
┌──────────────────────────────────────┐
│ Built-in (len, print, etc.) │
│ ┌───────────────────────────────┐ │
│ │ Global (module level) │ │
│ │ ┌────────────────────────┐ │ │
│ │ │ Enclosing (outer func) │ │ │
│ │ │ ┌─────────────────┐ │ │ │
│ │ │ │ Local (current) │ │ │ │
│ │ │ └─────────────────┘ │ │ │
│ │ └────────────────────────┘ │ │
│ └───────────────────────────────┘ │
└──────────────────────────────────────┘
x = "global" def outer(): x = "enclosing" def inner(): nonlocal x # Modify enclosing x = "modified" inner()
Basic Input/Output
Use print() for output with customizable separators and end characters; input() always returns a string, so cast for other types.
# Output print("Hello", "World", sep=", ", end="!\n") # Input name = input("Enter name: ") age = int(input("Enter age: ")) # Cast to int # Formatted output print(f"{'Name':<10} {'Age':>5}") # Aligned columns print(f"{'Alice':<10} {30:>5}")
Comments and Docstrings
Single-line comments use #; docstrings are triple-quoted strings immediately after function/class/module definitions, accessible via __doc__ and used by help systems.
# This is a single-line comment def calculate_area(radius): """ Calculate the area of a circle. Args: radius: The radius of the circle (float). Returns: The area as a float. """ return 3.14159 * radius ** 2 print(calculate_area.__doc__)
PEP 8 Style Guide
PEP 8 is Python's official style guide: use 4-space indentation, snake_case for functions/variables, PascalCase for classes, UPPER_CASE for constants, and limit lines to 79-88 characters.
# Good PEP 8 style MAX_CONNECTIONS = 100 # Constant class DatabaseConnection: # PascalCase def __init__(self): self.is_connected = False def connect_to_server(self): # snake_case pass def calculate_total(items): # snake_case return sum(items)
Basic Error Handling (try/except)
Wrap risky code in try blocks and handle specific exceptions in except clauses; use finally for cleanup that must always run.
try: result = 10 / 0 except ZeroDivisionError as e: print(f"Error: {e}") except (TypeError, ValueError) as e: print(f"Type/Value error: {e}") else: print("Success!") # Runs if no exception finally: print("Cleanup here") # Always runs
Importing Modules
Use import to access standard library and third-party modules; prefer explicit imports over from module import * for clarity and namespace control.
import os # Full module import numpy as np # With alias from pathlib import Path # Specific import from typing import List, Dict # Multiple imports from collections import defaultdict, Counter # Relative imports (in packages) from . import sibling_module from ..parent import something
File Reading and Writing
Always use context managers (with) for file operations to ensure proper resource cleanup; specify encoding explicitly for text files.
# Writing with open("data.txt", "w", encoding="utf-8") as f: f.write("Hello, World!\n") f.writelines(["Line 1\n", "Line 2\n"]) # Reading with open("data.txt", "r", encoding="utf-8") as f: content = f.read() # Entire file # or lines = f.readlines() # List of lines # or for line in f: # Memory efficient print(line.strip())
Working with Paths
Use pathlib.Path (Python 3.4+) instead of os.path for object-oriented, cross-platform path manipulation with cleaner syntax.
from pathlib import Path # Path operations path = Path("/home/user/docs") file_path = path / "report.txt" # Join with / file_path.exists() # Check existence file_path.is_file() # Is it a file? file_path.suffix # ".txt" file_path.stem # "report" file_path.parent # Path("/home/user/docs") # Globbing for py_file in Path(".").glob("**/*.py"): print(py_file)
ADVANCED
*args and **kwargs
*args collects positional arguments into a tuple, **kwargs collects keyword arguments into a dictionary; use them for flexible function signatures.
def log_message(level, *args, **kwargs): print(f"[{level}]", *args) for key, value in kwargs.items(): print(f" {key}: {value}") log_message("INFO", "Server started", port=8080, host="localhost") # [INFO] Server started # port: 8080 # host: localhost # Unpacking def add(a, b, c): return a + b + c nums = [1, 2, 3] add(*nums) # Unpack list: add(1, 2, 3)
Lambda Functions
Anonymous single-expression functions using lambda, useful for short callbacks; avoid for complex logic—use regular functions instead.
# Syntax: lambda arguments: expression square = lambda x: x ** 2 add = lambda x, y: x + y # Common use: sorting users = [("alice", 30), ("bob", 25), ("charlie", 35)] sorted(users, key=lambda u: u[1]) # Sort by age # With conditionals classify = lambda x: "positive" if x > 0 else "non-positive"
Map, Filter, Reduce
Functional programming utilities: map applies a function to all items, filter selects items matching a condition, reduce cumulatively combines items.
from functools import reduce nums = [1, 2, 3, 4, 5] # Map: apply function to each element squares = list(map(lambda x: x**2, nums)) # [1, 4, 9, 16, 25] # Filter: keep elements matching predicate evens = list(filter(lambda x: x % 2 == 0, nums)) # [2, 4] # Reduce: accumulate values total = reduce(lambda acc, x: acc + x, nums, 0) # 15 # Prefer list comprehensions for readability squares = [x**2 for x in nums] evens = [x for x in nums if x % 2 == 0]
Generators and Iterators
Generators produce values lazily using yield, consuming memory for only one item at a time; essential for processing large datasets.
# Generator function def countdown(n): while n > 0: yield n # Pause and return value n -= 1 for num in countdown(5): print(num) # 5, 4, 3, 2, 1 # Generator expression squares = (x**2 for x in range(1000000)) # No memory for all values first_ten = [next(squares) for _ in range(10)] # Iterator protocol class Counter: def __init__(self, max): self.max = max self.n = 0 def __iter__(self): return self def __next__(self): if self.n >= self.max: raise StopIteration self.n += 1 return self.n
Decorators (Basic)
Decorators are functions that wrap other functions to extend behavior without modifying the original code; use functools.wraps to preserve metadata.
from functools import wraps import time def timer(func): @wraps(func) # Preserve __name__, __doc__ def wrapper(*args, **kwargs): start = time.time() result = func(*args, **kwargs) print(f"{func.__name__} took {time.time() - start:.2f}s") return result return wrapper @timer # Equivalent: slow_function = timer(slow_function) def slow_function(): time.sleep(1) return "done" slow_function() # "slow_function took 1.00s"
Context Managers (with statement)
Context managers guarantee cleanup via __enter__ and __exit__ methods; essential for resources like files, locks, and database connections.
# Custom context manager (class-based) class DatabaseConnection: def __enter__(self): print("Opening connection") self.conn = "connection_object" return self.conn def __exit__(self, exc_type, exc_val, exc_tb): print("Closing connection") # Return True to suppress exception return False with DatabaseConnection() as conn: print(f"Using {conn}") # Output: # Opening connection # Using connection_object # Closing connection
Object-Oriented Programming (Classes, Objects)
Classes are blueprints for objects combining data (attributes) and behavior (methods); __init__ is the constructor, and self refers to the instance.
class Dog: species = "Canis familiaris" # Class attribute def __init__(self, name, age): self.name = name # Instance attribute self.age = age def bark(self): return f"{self.name} says woof!" def __str__(self): return f"Dog({self.name}, {self.age})" buddy = Dog("Buddy", 5) print(buddy.bark()) # "Buddy says woof!" print(Dog.species) # "Canis familiaris"
Inheritance and Polymorphism
Inheritance creates child classes that extend parent functionality; polymorphism allows different classes to be treated through a common interface.
class Animal: def __init__(self, name): self.name = name def speak(self): raise NotImplementedError class Dog(Animal): def speak(self): return f"{self.name}: Woof!" class Cat(Animal): def speak(self): return f"{self.name}: Meow!" # Polymorphism: same interface, different behavior animals = [Dog("Rex"), Cat("Whiskers")] for animal in animals: print(animal.speak()) # Rex: Woof! # Whiskers: Meow!
Encapsulation
Python uses naming conventions for access control: no underscore = public, single underscore _ = protected (internal use), double underscore __ = private (name-mangled).
class BankAccount: def __init__(self, balance): self._balance = balance # Protected self.__pin = "1234" # Private (name-mangled) def deposit(self, amount): if amount > 0: self._balance += amount def get_balance(self): return self._balance account = BankAccount(100) account._balance # Accessible (but discouraged) account.__pin # AttributeError! account._BankAccount__pin # Accessible via name mangling
Class Methods and Static Methods
@classmethod receives the class as first argument (cls) for factory patterns; @staticmethod receives no implicit argument and is basically a namespaced utility function.
class Date: def __init__(self, year, month, day): self.year = year self.month = month self.day = day @classmethod def from_string(cls, date_str): # Factory method year, month, day = map(int, date_str.split('-')) return cls(year, month, day) @staticmethod def is_valid_date(date_str): # Utility, no cls/self try: parts = date_str.split('-') return len(parts) == 3 except: return False date = Date.from_string("2024-01-15") # Uses classmethod Date.is_valid_date("2024-01-15") # True
Property Decorators
@property creates managed attributes with getter/setter/deleter methods, providing controlled access while maintaining attribute-like syntax.
class Circle: def __init__(self, radius): self._radius = radius @property def radius(self): return self._radius @radius.setter def radius(self, value): if value < 0: raise ValueError("Radius cannot be negative") self._radius = value @property def area(self): # Computed property return 3.14159 * self._radius ** 2 c = Circle(5) print(c.radius) # 5 (uses getter) c.radius = 10 # Uses setter print(c.area) # 314.159 (computed)
Magic Methods (Dunder Methods)
Double-underscore methods like __init__, __str__, __add__ enable operator overloading and Python's object protocols.
class Vector: def __init__(self, x, y): self.x, self.y = x, y def __repr__(self): # Developer representation return f"Vector({self.x}, {self.y})" def __str__(self): # User-friendly string return f"({self.x}, {self.y})" def __add__(self, other): # + operator return Vector(self.x + other.x, self.y + other.y) def __eq__(self, other): # == operator return self.x == other.x and self.y == other.y def __len__(self): # len() function return int((self.x**2 + self.y**2)**0.5) v1 = Vector(3, 4) v2 = Vector(1, 2) print(v1 + v2) # (4, 6) print(len(v1)) # 5
Multiple Inheritance and MRO
Python supports multiple inheritance; Method Resolution Order (MRO) uses C3 linearization to determine which method to call—inspect with Class.__mro__.
┌─────────┐
│ object │
└────┬────┘
┌─────┴─────┐
┌────┴────┐ ┌────┴────┐
│ A │ │ B │
└────┬────┘ └────┬────┘
└─────┬─────┘
┌────┴────┐
│ C │ MRO: C → A → B → object
└─────────┘
class A: def greet(self): return "A" class B: def greet(self): return "B" class C(A, B): # A comes first pass print(C().greet()) # "A" (A checked before B) print(C.__mro__) # (C, A, B, object)
Abstract Base Classes
ABCs define interfaces that subclasses must implement; use abc module to create abstract methods that enforce implementation.
from abc import ABC, abstractmethod class Shape(ABC): @abstractmethod def area(self): pass @abstractmethod def perimeter(self): pass class Rectangle(Shape): def __init__(self, width, height): self.width = width self.height = height def area(self): # Must implement return self.width * self.height def perimeter(self): # Must implement return 2 * (self.width + self.height) # Shape() # TypeError: Can't instantiate abstract class rect = Rectangle(4, 5) # OK
Regular Expressions
The re module provides Perl-style regex for pattern matching; compile patterns for reuse and use raw strings (r"") to avoid escape issues.
import re text = "Contact: alice@google.com or bob@example.org" # Find all emails pattern = r'[\w.+-]+@[\w-]+\.[\w.-]+' emails = re.findall(pattern, text) # ['alice@google.com', 'bob@example.org'] # Search and groups match = re.search(r'(\w+)@(\w+)\.(\w+)', text) if match: print(match.group(0)) # alice@google.com print(match.group(1)) # alice # Substitution cleaned = re.sub(r'\d+', 'XXX', 'Order 12345') # "Order XXX" # Compile for reuse email_re = re.compile(r'[\w.+-]+@[\w-]+\.[\w.-]+') email_re.findall(another_text)
JSON Handling
Use the json module for serialization; dumps/loads work with strings, dump/load work with files.
import json data = {"name": "Alice", "age": 30, "active": True} # Serialize to string json_str = json.dumps(data, indent=2) print(json_str) # Parse from string parsed = json.loads(json_str) # File operations with open("data.json", "w") as f: json.dump(data, f, indent=2) with open("data.json", "r") as f: loaded = json.load(f) # Custom serialization class User: def __init__(self, name): self.name = name json.dumps(user, default=lambda o: o.__dict__)
CSV Handling
The csv module handles comma-separated values with reader/writer for lists and DictReader/DictWriter for dictionary-based row access.
import csv # Writing CSV with open("users.csv", "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=["name", "age"]) writer.writeheader() writer.writerow({"name": "Alice", "age": 30}) writer.writerow({"name": "Bob", "age": 25}) # Reading CSV with open("users.csv", "r") as f: reader = csv.DictReader(f) for row in reader: print(row["name"], row["age"]) # Simple list-based with open("data.csv", "r") as f: reader = csv.reader(f) for row in reader: print(row) # ['col1', 'col2', ...]
Date and Time Operations
Use datetime module for date/time manipulation; prefer timezone-aware objects for production code and ISO format for serialization.
from datetime import datetime, timedelta, timezone # Current time now = datetime.now() # Local time utc_now = datetime.now(timezone.utc) # UTC # Parsing and formatting dt = datetime.strptime("2024-01-15", "%Y-%m-%d") formatted = dt.strftime("%B %d, %Y") # "January 15, 2024" # Arithmetic tomorrow = now + timedelta(days=1) diff = datetime(2024, 12, 31) - now print(diff.days) # ISO format (recommended for APIs) iso_str = now.isoformat() # "2024-01-15T10:30:00" parsed = datetime.fromisoformat(iso_str)
Collections Module
Extended container types: defaultdict for auto-initialized values, Counter for counting, deque for fast appends/pops on both ends, namedtuple for lightweight records.
from collections import defaultdict, Counter, deque, OrderedDict # defaultdict: auto-initializes missing keys word_counts = defaultdict(int) for word in ["a", "b", "a"]: word_counts[word] += 1 # {"a": 2, "b": 1} # Counter: count occurrences counts = Counter("mississippi") # {'i': 4, 's': 4, 'p': 2, 'm': 1} counts.most_common(2) # [('i', 4), ('s', 4)] # deque: O(1) operations on both ends q = deque([1, 2, 3], maxlen=5) q.appendleft(0) # deque([0, 1, 2, 3]) q.pop() # 3
Itertools Module
A toolkit for efficient iteration: chain, cycle, repeat, combinations, permutations, groupby, and more.
from itertools import chain, cycle, islice, groupby, combinations, product # chain: flatten iterables list(chain([1, 2], [3, 4])) # [1, 2, 3, 4] # combinations and permutations list(combinations([1, 2, 3], 2)) # [(1,2), (1,3), (2,3)] # product: cartesian product list(product("AB", [1, 2])) # [('A',1), ('A',2), ('B',1), ('B',2)] # groupby: group consecutive elements data = [("a", 1), ("a", 2), ("b", 3)] for key, group in groupby(data, key=lambda x: x[0]): print(key, list(group)) # islice: slice iterators list(islice(cycle([1, 2, 3]), 7)) # [1, 2, 3, 1, 2, 3, 1]
Functools Module
Higher-order functions: lru_cache for memoization, partial for pre-filling arguments, reduce for accumulation, wraps for decorator metadata.
from functools import lru_cache, partial, reduce, wraps # lru_cache: memoization @lru_cache(maxsize=128) def fibonacci(n): if n < 2: return n return fibonacci(n-1) + fibonacci(n-2) fibonacci(100) # Instant with caching # partial: pre-fill arguments def power(base, exp): return base ** exp square = partial(power, exp=2) cube = partial(power, exp=3) print(square(5)) # 25 # reduce: accumulate reduce(lambda a, b: a * b, [1, 2, 3, 4]) # 24
Exception Hierarchy
All exceptions inherit from BaseException; catch Exception for most errors but not KeyboardInterrupt or SystemExit.
BaseException
├── SystemExit
├── KeyboardInterrupt
├── GeneratorExit
└── Exception
├── StopIteration
├── ArithmeticError
│ ├── ZeroDivisionError
│ └── OverflowError
├── LookupError
│ ├── KeyError
│ └── IndexError
├── OSError (IOError, FileNotFoundError...)
├── ValueError
├── TypeError
└── RuntimeError
Custom Exceptions
Create application-specific exceptions by inheriting from Exception; include meaningful attributes and messages.
class ValidationError(Exception): """Raised when data validation fails.""" def __init__(self, field, message, value=None): self.field = field self.value = value super().__init__(f"{field}: {message}") class APIError(Exception): def __init__(self, status_code, message): self.status_code = status_code super().__init__(f"[{status_code}] {message}") # Usage def validate_age(age): if age < 0: raise ValidationError("age", "must be positive", age) try: validate_age(-5) except ValidationError as e: print(f"Field '{e.field}' invalid: {e}")
Logging Basics
Use the logging module instead of print statements; configure levels (DEBUG, INFO, WARNING, ERROR, CRITICAL) and handlers for flexible output.
import logging # Basic configuration logging.basicConfig( level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', handlers=[ logging.FileHandler('app.log'), logging.StreamHandler() ] ) logger = logging.getLogger(__name__) logger.debug("Debug info") # Won't show (level=INFO) logger.info("Server started") # Shows logger.warning("High memory usage") # Shows logger.error("Connection failed") # Shows logger.exception("Error with traceback") # Includes stack trace
Virtual Environments (venv, virtualenv)
Isolated Python environments prevent dependency conflicts between projects; venv is built-in since Python 3.3.
# Create virtual environment python -m venv .venv # Activate source .venv/bin/activate # Linux/macOS .venv\Scripts\activate # Windows # Verify which python # Should show .venv path pip list # Should be minimal # Install packages pip install requests # Deactivate deactivate
project/
├── .venv/ # Virtual environment (git-ignored)
├── src/
├── requirements.txt
└── .gitignore # Include: .venv/
pip and Package Management
pip is Python's package installer; use it to install, upgrade, and manage third-party packages from PyPI.
# Install packages pip install requests # Latest version pip install requests==2.28.0 # Specific version pip install "requests>=2.20,<3.0" # Version range pip install package[extra] # With optional dependencies pip install -e . # Editable install (development) # Upgrade pip install --upgrade requests # Uninstall pip uninstall requests # List installed pip list pip show requests # Package details # Export dependencies pip freeze > requirements.txt
requirements.txt
A file listing project dependencies for reproducible installs; pin versions in production, use ranges for libraries.
# requirements.txt requests==2.31.0 flask>=2.0,<3.0 numpy~=1.24.0 # Compatible release (~=1.24.0 means >=1.24.0,<1.25.0) pytest>=7.0 # Development dependencies (often separate: requirements-dev.txt) black==23.9.1 mypy==1.5.1 # From git # git+https://github.com/user/repo.git@v1.0.0#egg=package # Install from file # pip install -r requirements.txt
Type Hints (Basic)
Type annotations improve code readability and enable static analysis; they don't affect runtime behavior.
from typing import List, Dict, Optional, Union, Tuple def greet(name: str) -> str: return f"Hello, {name}" def process( items: List[int], config: Dict[str, str], threshold: Optional[float] = None # Can be None ) -> Tuple[int, bool]: return (len(items), threshold is not None) # Variables count: int = 0 names: List[str] = [] mapping: Dict[str, int] = {} # Union types (multiple possible types) def parse(value: Union[str, int]) -> str: return str(value) # Python 3.10+ syntax def parse(value: str | int) -> str: return str(value)
Enumeration
Enum creates named constants with meaningful names instead of magic values; use auto() for automatic value assignment.
from enum import Enum, auto, IntEnum class Status(Enum): PENDING = "pending" ACTIVE = "active" COMPLETED = "completed" @classmethod def from_string(cls, s): return cls(s.lower()) class Priority(IntEnum): # Can be compared with integers LOW = auto() # 1 MEDIUM = auto() # 2 HIGH = auto() # 3 # Usage status = Status.ACTIVE print(status.name) # "ACTIVE" print(status.value) # "active" if status == Status.ACTIVE: print("Task is active") # Iteration for s in Status: print(s)
Metaclasses
Metaclasses are "classes of classes" that control class creation; type is the default metaclass, and custom metaclasses can modify class behavior at definition time.
class SingletonMeta(type): _instances = {} def __call__(cls, *args, **kwargs): if cls not in cls._instances: cls._instances[cls] = super().__call__(*args, **kwargs) return cls._instances[cls] class Database(metaclass=SingletonMeta): def __init__(self): self.connection = "connected" db1 = Database() db2 = Database() print(db1 is db2) # True - same instance
Descriptors
Descriptors implement __get__, __set__, or __delete__ to control attribute access; they power @property, @classmethod, and @staticmethod.
class Validated: def __init__(self, min_value=None, max_value=None): self.min_value = min_value self.max_value = max_value def __set_name__(self, owner, name): self.name = name def __get__(self, obj, objtype=None): if obj is None: return self return obj.__dict__.get(self.name) def __set__(self, obj, value): if self.min_value is not None and value < self.min_value: raise ValueError(f"{self.name} must be >= {self.min_value}") obj.__dict__[self.name] = value class Order: quantity = Validated(min_value=0) price = Validated(min_value=0) order = Order() order.quantity = 10 # OK order.quantity = -1 # ValueError
Advanced Decorators (with Parameters, Class Decorators)
Parameterized decorators use a wrapper factory pattern; class decorators modify or wrap entire classes.
from functools import wraps # Decorator with parameters def retry(max_attempts=3, delay=1): def decorator(func): @wraps(func) def wrapper(*args, **kwargs): for attempt in range(max_attempts): try: return func(*args, **kwargs) except Exception as e: if attempt == max_attempts - 1: raise time.sleep(delay) return wrapper return decorator @retry(max_attempts=5, delay=2) def fetch_data(): pass # Class decorator def singleton(cls): instances = {} @wraps(cls) def get_instance(*args, **kwargs): if cls not in instances: instances[cls] = cls(*args, **kwargs) return instances[cls] return get_instance @singleton class Config: pass
Closures
A closure captures variables from its enclosing scope, retaining access to them even after the outer function has finished executing.
def make_multiplier(factor): # 'factor' is captured in the closure def multiply(x): return x * factor return multiply double = make_multiplier(2) triple = make_multiplier(3) print(double(5)) # 10 print(triple(5)) # 15 # Inspecting closure print(double.__closure__) # (<cell at 0x...: int object at 0x...>,) print(double.__closure__[0].cell_contents) # 2 # Mutable closure with nonlocal def counter(): count = 0 def increment(): nonlocal count count += 1 return count return increment c = counter() print(c(), c(), c()) # 1 2 3
Coroutines
Coroutines are functions that can be paused and resumed, enabling cooperative multitasking; modern Python uses async def syntax.
# Generator-based coroutine (legacy) def coroutine(): value = None while True: received = yield value value = received * 2 coro = coroutine() next(coro) # Prime the coroutine print(coro.send(5)) # 10 print(coro.send(10)) # 20 # Modern async coroutine async def async_task(): await asyncio.sleep(1) return "done" # Must be run in event loop # asyncio.run(async_task())
Async/Await Basics
async def creates a coroutine, await pauses execution until the awaitable completes; enables concurrent I/O operations without threads.
import asyncio async def fetch_data(url: str, delay: float) -> str: print(f"Fetching {url}...") await asyncio.sleep(delay) # Non-blocking wait return f"Data from {url}" async def main(): # Sequential (slow) result1 = await fetch_data("url1", 1) result2 = await fetch_data("url2", 1) # Concurrent (fast) results = await asyncio.gather( fetch_data("url1", 1), fetch_data("url2", 1), fetch_data("url3", 1), ) print(results) asyncio.run(main())
Asyncio Fundamentals
asyncio provides the event loop, tasks, and synchronization primitives for async programming.
import asyncio async def worker(name: str, queue: asyncio.Queue): while True: item = await queue.get() print(f"{name} processing {item}") await asyncio.sleep(0.5) queue.task_done() async def main(): queue = asyncio.Queue() # Create tasks workers = [asyncio.create_task(worker(f"Worker-{i}", queue)) for i in range(3)] # Add items for i in range(10): await queue.put(f"item-{i}") await queue.join() # Wait for all items # Cleanup for w in workers: w.cancel() asyncio.run(main())
Concurrent Programming (Threading)
Threads allow concurrent execution but share memory; due to the GIL, they're best for I/O-bound tasks, not CPU-bound.
import threading from concurrent.futures import ThreadPoolExecutor # Basic threading def worker(name): print(f"Thread {name} starting") time.sleep(1) print(f"Thread {name} finished") thread = threading.Thread(target=worker, args=("A",)) thread.start() thread.join() # Wait for completion # Thread pool (preferred) def fetch_url(url): return f"Fetched {url}" with ThreadPoolExecutor(max_workers=5) as executor: urls = ["url1", "url2", "url3"] results = list(executor.map(fetch_url, urls)) # Or with futures future = executor.submit(fetch_url, "url4") result = future.result(timeout=5)
Multiprocessing
Multiprocessing bypasses the GIL by running separate Python processes; ideal for CPU-bound tasks.
from multiprocessing import Pool, Process, Queue import os def cpu_intensive(n): return sum(i * i for i in range(n)) # Process pool if __name__ == "__main__": with Pool(processes=4) as pool: results = pool.map(cpu_intensive, [10**6, 10**6, 10**6, 10**6]) print(results) # Async with callback async_result = pool.apply_async(cpu_intensive, (10**6,)) result = async_result.get(timeout=10) # Inter-process communication def producer(queue): queue.put("data from producer") if __name__ == "__main__": q = Queue() p = Process(target=producer, args=(q,)) p.start() print(q.get()) p.join()
GIL (Global Interpreter Lock)
The GIL is a mutex in CPython that prevents multiple threads from executing Python bytecode simultaneously, making threads unsuitable for CPU-bound parallelism.
┌────────────────────────────────────────────────────────────┐
│ GIL Impact │
├────────────────────────────────────────────────────────────┤
│ │
│ Threading (with GIL): Thread 1 ████░░░░████░░░░ │
│ Thread 2 ░░░░████░░░░████ │
│ (Takes turns - no true parallel)│
│ │
│ Multiprocessing: Process 1 ████████████████ │
│ Process 2 ████████████████ │
│ (True parallel execution) │
│ │
├────────────────────────────────────────────────────────────┤
│ Use threading for: I/O-bound tasks (network, file, DB) │
│ Use multiprocessing for: CPU-bound tasks (computation) │
└────────────────────────────────────────────────────────────┘
Memory Management
Python uses reference counting plus a cyclic garbage collector; understanding memory helps optimize performance.
import sys a = [1, 2, 3] print(sys.getsizeof(a)) # Size in bytes # Reference counting b = a # refcount = 2 print(sys.getrefcount(a)) # 3 (includes function arg) del b # refcount = 1 a = None # refcount = 0, object freed # Memory-efficient patterns # Use generators instead of lists for large sequences gen = (x**2 for x in range(1000000)) # Minimal memory # Use __slots__ for many instances class Point: __slots__ = ['x', 'y'] def __init__(self, x, y): self.x, self.y = x, y
Garbage Collection
Python's garbage collector handles reference cycles that reference counting can't free; controllable via the gc module.
import gc # Manual control gc.disable() # Disable auto collection gc.enable() # Re-enable gc.collect() # Force collection # Get collection stats print(gc.get_stats()) # Find reference cycles gc.set_debug(gc.DEBUG_SAVEALL) gc.collect() print(gc.garbage) # Uncollectable objects # Weak references avoid cycles import weakref class Cache: def __init__(self): self._cache = weakref.WeakValueDictionary() def get(self, key): return self._cache.get(key)
Weak References
Weak references allow referencing an object without preventing its garbage collection; useful for caches and observer patterns.
import weakref class ExpensiveObject: def __init__(self, value): self.value = value def __del__(self): print(f"Deleting {self.value}") obj = ExpensiveObject("data") weak_ref = weakref.ref(obj) print(weak_ref()) # <ExpensiveObject object> print(weak_ref().value) # "data" del obj # "Deleting data" - object freed print(weak_ref()) # None # WeakValueDictionary for caches cache = weakref.WeakValueDictionary() obj = ExpensiveObject("cached") cache["key"] = obj del obj # Automatically removed from cache
__slots__ Optimization
__slots__ replaces the instance __dict__ with a fixed-size array, reducing memory usage by 40-50% for many small objects.
import sys class RegularPoint: def __init__(self, x, y): self.x = x self.y = y class SlottedPoint: __slots__ = ['x', 'y'] def __init__(self, x, y): self.x = x self.y = y regular = RegularPoint(1, 2) slotted = SlottedPoint(1, 2) print(sys.getsizeof(regular.__dict__)) # ~104 bytes for dict # slotted has no __dict__ # Memory comparison with 1 million instances: # Regular: ~200 MB # Slotted: ~80 MB # Limitations: # - No dynamic attributes # - Must be defined in each class (not inherited by default) slotted.z = 3 # AttributeError!
Context Managers (contextlib)
The contextlib module provides utilities for creating context managers without boilerplate class definitions.
from contextlib import contextmanager, suppress, ExitStack # Function-based context manager @contextmanager def timer(name): start = time.time() try: yield # Control transfers to with-block finally: print(f"{name}: {time.time() - start:.2f}s") with timer("operation"): time.sleep(1) # Suppress specific exceptions with suppress(FileNotFoundError): os.remove("nonexistent.txt") # No error raised # Dynamic context manager stacking with ExitStack() as stack: files = [stack.enter_context(open(f)) for f in filenames] # All files closed on exit
Advanced Generators (send, throw, close)
Generators can receive values via send(), handle exceptions via throw(), and clean up via close().
def accumulator(): total = 0 while True: try: value = yield total if value is None: break total += value except ValueError: print("Invalid value, skipping") finally: pass gen = accumulator() next(gen) # Prime: returns 0 print(gen.send(10)) # 10 print(gen.send(20)) # 30 gen.throw(ValueError) # "Invalid value, skipping", returns 30 gen.close() # Triggers GeneratorExit # Pipeline pattern def pipeline(source, *transforms): for item in source: for transform in transforms: item = transform(item) yield item
Generator Expressions vs List Comprehensions
Generator expressions use parentheses and are lazy (memory-efficient); list comprehensions use brackets and create the full list immediately.
# List comprehension - creates full list in memory squares_list = [x**2 for x in range(1000000)] # ~8 MB memory # Generator expression - lazy, minimal memory squares_gen = (x**2 for x in range(1000000)) # ~120 bytes # Performance comparison import sys print(sys.getsizeof(squares_list)) # 8448728 print(sys.getsizeof(squares_gen)) # 112 # Use cases: # List: need random access, multiple iterations, len() # Generator: single iteration, large data, chaining # Generator is faster for "any/all" short-circuits any(x > 1000 for x in range(1000000)) # Stops at 1001 # Can only iterate generator once! list(squares_gen) # Works list(squares_gen) # Empty!
Data Classes
@dataclass generates boilerplate (__init__, __repr__, __eq__, etc.) automatically for data-holding classes.
from dataclasses import dataclass, field, asdict from typing import List @dataclass class User: name: str email: str age: int = 0 tags: List[str] = field(default_factory=list) def __post_init__(self): self.email = self.email.lower() @dataclass(frozen=True) # Immutable (hashable) class Point: x: float y: float user = User("Alice", "Alice@Gmail.com", 30) print(user) # User(name='Alice', email='alice@gmail.com', ...) print(asdict(user)) # {'name': 'Alice', ...} # Comparison and hashing p1 = Point(1, 2) p2 = Point(1, 2) print(p1 == p2) # True print({p1, p2}) # {Point(x=1, y=2)} - deduped
Named Tuples
Named tuples are immutable, memory-efficient records with named fields; use typing.NamedTuple for type hints.
from typing import NamedTuple from collections import namedtuple # Class syntax (preferred) class Point(NamedTuple): x: float y: float label: str = "origin" # Factory function Point = namedtuple('Point', ['x', 'y', 'label']) p = Point(3, 4, "A") print(p.x, p.y) # 3 4 print(p[0], p[1]) # 3 4 (tuple access) x, y, label = p # Unpacking # Immutable p.x = 5 # AttributeError! # Convert to dict p._asdict() # {'x': 3, 'y': 4, 'label': 'A'} # Create modified copy p2 = p._replace(x=10) # Point(10, 4, 'A')
Type Hints (Advanced, Generics)
Advanced typing enables generic classes, protocols, and complex type relationships for robust static analysis.
from typing import TypeVar, Generic, Callable, overload, Literal T = TypeVar('T') K = TypeVar('K') V = TypeVar('V') class Stack(Generic[T]): def __init__(self) -> None: self._items: list[T] = [] def push(self, item: T) -> None: self._items.append(item) def pop(self) -> T: return self._items.pop() stack: Stack[int] = Stack() stack.push(1) stack.push("str") # mypy error! # Callable types Handler = Callable[[str, int], bool] def process(handler: Handler) -> None: handler("test", 42) # Literal types Mode = Literal["r", "w", "a"] def open_file(path: str, mode: Mode) -> None: pass
Protocol Classes
Protocols enable structural subtyping (duck typing with type checking); classes don't need to inherit from a Protocol to match it.
from typing import Protocol, runtime_checkable @runtime_checkable class Drawable(Protocol): def draw(self) -> str: ... class Circle: # No inheritance needed! def draw(self) -> str: return "Drawing circle" class Square: def draw(self) -> str: return "Drawing square" def render(shape: Drawable) -> None: # Accepts anything with draw() print(shape.draw()) render(Circle()) # OK render(Square()) # OK # Runtime check print(isinstance(Circle(), Drawable)) # True
Structural Pattern Matching (Python 3.10+)
match/case statements enable powerful pattern matching on structure, not just value equality.
def process_command(command): match command: case ["quit"]: return "Exiting" case ["load", filename]: return f"Loading {filename}" case ["save", filename, ("json" | "csv") as fmt]: return f"Saving {filename} as {fmt}" case {"action": "create", "name": name, **rest}: return f"Creating {name} with {rest}" case [*items] if len(items) > 3: return f"Too many items: {items}" case _: return "Unknown command" print(process_command(["load", "data.txt"])) print(process_command({"action": "create", "name": "test", "type": "user"}))
Walrus Operator
The walrus operator := assigns values as part of an expression, reducing code duplication and improving readability.
# Before line = input() while line != "quit": print(line) line = input() # After while (line := input()) != "quit": print(line) # In list comprehensions results = [y for x in data if (y := expensive_func(x)) > threshold] # In conditions if (match := re.search(pattern, text)): print(match.group(0)) # File reading while (chunk := file.read(8192)): process(chunk) # With any/all if any((n := x) > 10 for x in numbers): print(f"Found: {n}")
F-strings Advanced Features
F-strings support expressions, formatting specs, debugging output (=), and nesting.
name = "Alice" value = 123.456789 # Alignment and width f"|{name:<10}|" # |Alice | f"|{name:>10}|" # | Alice| f"|{name:^10}|" # | Alice | # Number formatting f"{value:.2f}" # "123.46" f"{value:,.2f}" # "123.46" (with comma separator) f"{1000000:_}" # "1_000_000" f"{255:#x}" # "0xff" f"{0.25:.1%}" # "25.0%" # Debug mode (Python 3.8+) x, y = 10, 20 f"{x=}, {y=}" # "x=10, y=20" f"{x + y = }" # "x + y = 30" # Nesting f"{value:.{precision}f}" # Dynamic precision # DateTime formatting from datetime import datetime now = datetime.now() f"{now:%Y-%m-%d %H:%M}" # "2024-01-15 10:30"
Packaging (setup.py, setup.cfg, pyproject.toml)
pyproject.toml is the modern standard for Python packaging configuration, replacing legacy setup.py.
# pyproject.toml [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [project] name = "mypackage" version = "0.1.0" description = "My awesome package" readme = "README.md" requires-python = ">=3.8" dependencies = [ "requests>=2.28", "click>=8.0", ] [project.optional-dependencies] dev = ["pytest", "black", "mypy"] [project.scripts] mycommand = "mypackage.cli:main" [tool.black] line-length = 88 [tool.pytest.ini_options] testpaths = ["tests"]
Poetry
Poetry is a modern dependency management and packaging tool that handles virtual environments, dependency resolution, and publishing.
# Install curl -sSL https://install.python-poetry.org | python3 - # New project poetry new myproject poetry init # Interactive setup in existing dir # Dependency management poetry add requests # Add dependency poetry add pytest --group dev # Dev dependency poetry remove requests # Remove poetry update # Update all # Virtual environment poetry install # Install all deps poetry shell # Activate venv poetry run pytest # Run in venv # Build and publish poetry build # Creates wheel and sdist poetry publish # Upload to PyPI
Building Distributions (wheel, sdist)
Source distributions (sdist) contain source code; wheels are pre-built binaries for faster installation.
# Build distributions pip install build python -m build # Creates dist/ # Output: # dist/ # ├── mypackage-0.1.0.tar.gz # sdist # └── mypackage-0.1.0-py3-none-any.whl # wheel # Install from wheel (fast) pip install dist/mypackage-0.1.0-py3-none-any.whl # Upload to PyPI pip install twine twine upload dist/* # Test PyPI first twine upload --repository testpypi dist/* pip install --index-url https://test.pypi.org/simple/ mypackage
Wheel filename format:
{package}-{version}-{python}-{abi}-{platform}.whl
Examples:
numpy-1.24.0-cp311-cp311-linux_x86_64.whl # Platform-specific
requests-2.31.0-py3-none-any.whl # Pure Python
Testing with unittest
Python's built-in testing framework provides test organization, assertions, and fixtures through class-based tests.
import unittest from mymodule import Calculator class TestCalculator(unittest.TestCase): @classmethod def setUpClass(cls): # Once per class cls.shared_resource = "db_connection" def setUp(self): # Before each test self.calc = Calculator() def tearDown(self): # After each test pass def test_add(self): self.assertEqual(self.calc.add(2, 3), 5) def test_divide_by_zero(self): with self.assertRaises(ZeroDivisionError): self.calc.divide(1, 0) @unittest.skip("Not implemented yet") def test_future_feature(self): pass if __name__ == '__main__': unittest.main()
Testing with pytest
pytest is the de-facto testing standard, using simple functions, powerful fixtures, and extensive plugin ecosystem.
# test_example.py import pytest from mymodule import Calculator def test_add(): calc = Calculator() assert calc.add(2, 3) == 5 def test_divide(): calc = Calculator() assert calc.divide(10, 2) == 5 def test_divide_by_zero(): calc = Calculator() with pytest.raises(ZeroDivisionError): calc.divide(1, 0) class TestGroup: def test_in_class(self): assert True # Run with: pytest -v # Coverage: pytest --cov=mymodule
Fixtures and Parametrization
Fixtures provide reusable test setup/teardown; parametrization runs tests with multiple inputs.
import pytest @pytest.fixture def calculator(): """Provides a Calculator instance.""" calc = Calculator() yield calc # Setup complete calc.cleanup() # Teardown @pytest.fixture(scope="module") # Once per module def database(): db = connect_db() yield db db.close() def test_add(calculator): # Fixture auto-injected assert calculator.add(1, 2) == 3 @pytest.mark.parametrize("a,b,expected", [ (1, 2, 3), (0, 0, 0), (-1, 1, 0), (100, 200, 300), ]) def test_add_parametrized(calculator, a, b, expected): assert calculator.add(a, b) == expected
Mocking and Patching
unittest.mock replaces dependencies with controllable doubles; use @patch decorator or context managers.
from unittest.mock import Mock, patch, MagicMock # Basic mock mock_db = Mock() mock_db.query.return_value = [{"id": 1, "name": "Alice"}] result = mock_db.query("SELECT *") mock_db.query.assert_called_once_with("SELECT *") # Patching class UserService: def get_user(self, user_id): return requests.get(f"/users/{user_id}").json() @patch('mymodule.requests.get') def test_get_user(mock_get): mock_get.return_value.json.return_value = {"id": 1, "name": "Alice"} service = UserService() user = service.get_user(1) assert user["name"] == "Alice" mock_get.assert_called_with("/users/1") # Context manager style with patch.object(SomeClass, 'method') as mock_method: mock_method.return_value = "mocked"
Code Coverage
Coverage measures which code is executed during tests; aim for 80%+ in production code.
# Install pip install pytest-cov coverage # Run with coverage pytest --cov=mypackage --cov-report=html tests/ # Output # Name Stmts Miss Cover # ---------------------------------------- # mypackage/__init__ 5 0 100% # mypackage/core.py 50 10 80% # ---------------------------------------- # TOTAL 55 10 82% # HTML report open htmlcov/index.html # Fail if coverage below threshold pytest --cov=mypackage --cov-fail-under=80
# .coveragerc or pyproject.toml [coverage:run] omit = tests/* */__init__.py [coverage:report] exclude_lines = pragma: no cover if TYPE_CHECKING:
Profiling (cProfile, line_profiler)
Profilers identify performance bottlenecks; cProfile for function-level, line_profiler for line-by-line analysis.
# cProfile - function level import cProfile import pstats cProfile.run('my_function()', 'output.prof') # Analyze stats = pstats.Stats('output.prof') stats.sort_stats('cumulative').print_stats(10) # Command line # python -m cProfile -s cumulative script.py # line_profiler - line by line # pip install line_profiler @profile # Decorator (when running with kernprof) def slow_function(): result = [] for i in range(10000): result.append(i ** 2) return result # Run: kernprof -l -v script.py
Output example:
ncalls tottime percall cumtime percall filename:lineno(function)
1000 0.234 0.000 1.345 0.001 mymodule.py:10(process)
100 0.567 0.006 0.567 0.006 mymodule.py:25(calculate)
Memory Profiling
Memory profilers track allocation and identify leaks; essential for long-running applications.
# pip install memory_profiler from memory_profiler import profile @profile def memory_intensive(): a = [1] * (10 ** 6) # ~8 MB b = [2] * (2 * 10 ** 7) # ~160 MB del b return a # Run: python -m memory_profiler script.py
Output:
Line # Mem usage Increment Line Contents
================================================
3 38.0 MiB 38.0 MiB @profile
4 def memory_intensive():
5 45.6 MiB 7.6 MiB a = [1] * (10 ** 6)
6 198.4 MiB 152.8 MiB b = [2] * (2 * 10 ** 7)
7 45.6 MiB -152.8 MiB del b
8 45.6 MiB 0.0 MiB return a
# tracemalloc (built-in) import tracemalloc tracemalloc.start() # ... code ... snapshot = tracemalloc.take_snapshot() top_stats = snapshot.statistics('lineno')
Debugging (pdb, ipdb)
pdb is Python's built-in debugger; ipdb adds IPython features for a better experience.
# Insert breakpoint def buggy_function(x): result = x * 2 breakpoint() # Python 3.7+ (or: import pdb; pdb.set_trace()) result += mysterious_calc(result) return result # pdb commands: # n(ext) - execute next line # s(tep) - step into function # c(ontinue) - continue to next breakpoint # p expr - print expression # pp expr - pretty print # l(ist) - show current code # w(here) - show stack trace # q(uit) - exit debugger # b 42 - set breakpoint at line 42 # b func - break at function entry # Post-mortem debugging python -m pdb script.py # Start with debugger python -m pdb -c continue script.py # Run until exception # ipdb (better REPL) pip install ipdb import ipdb; ipdb.set_trace()
Static Type Checking (mypy)
mypy analyzes type hints at development time, catching type errors before runtime.
# Install pip install mypy # Run mypy mypackage/ mypy --strict script.py # Strict mode
# mypy catches these errors: def greet(name: str) -> str: return f"Hello, {name}" greet(123) # error: Argument 1 has incompatible type "int" # Configuration in pyproject.toml [tool.mypy] python_version = "3.11" warn_return_any = true warn_unused_ignores = true disallow_untyped_defs = true [[tool.mypy.overrides]] module = "third_party.*" ignore_missing_imports = true
mypy output:
script.py:10: error: Argument 1 to "greet" has incompatible type "int"; expected "str"
script.py:15: error: Function is missing a return type annotation
Found 2 errors in 1 file
Linting (pylint, flake8, black)
Linters enforce code quality and style consistency; use them together for comprehensive coverage.
# Flake8: style checker (PEP 8 + logical errors) pip install flake8 flake8 mypackage/ # Pylint: comprehensive linter (more checks) pip install pylint pylint mypackage/ # Black: opinionated formatter (auto-fixes) pip install black black mypackage/ # Format in place black --check mypackage/ # Check only # isort: import sorter pip install isort isort mypackage/ # Ruff: fast all-in-one (modern choice) pip install ruff ruff check mypackage/ ruff format mypackage/
# pyproject.toml [tool.black] line-length = 88 target-version = ['py311'] [tool.ruff] line-length = 88 select = ["E", "F", "W", "I"] [tool.pylint.messages_control] disable = ["C0114", "C0115"]
Pre-commit Hooks
Pre-commit runs checks automatically before each commit, ensuring consistent code quality across the team.
# .pre-commit-config.yaml repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.5.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - id: check-yaml - id: check-added-large-files - repo: https://github.com/psf/black rev: 23.9.1 hooks: - id: black - repo: https://github.com/charliermarsh/ruff-pre-commit rev: v0.1.0 hooks: - id: ruff - repo: https://github.com/pre-commit/mirrors-mypy rev: v1.5.1 hooks: - id: mypy additional_dependencies: [types-requests]
# Setup pip install pre-commit pre-commit install # Install git hooks # Usage (automatic on commit) git commit -m "feat: add feature" # Hooks run automatically # Manual run pre-commit run --all-files