Lesson 1 of 8 · 8 min · Working with tables in pandas
DataFrames and Series: loading and inspecting data
What you will learn
- What a
DataFrameand aSeriesare - Load a CSV or Excel file and inspect it
- Select columns and create new ones
- Understand data types and the index
- Save results to a file
In simple words
pandas is a Python tool for working with tables. A DataFrame is a table in your program: rows and columns with names, like an Excel sheet or a SQL table. A Series is just one column of it. You load a file into a DataFrame, look at it, and then ask questions of it in a line or two of code, instead of clicking through thousands of cells.
Words to know
- DataFrame: a table of rows and columns in pandas.
- Series: a single column of a DataFrame, with an index.
- Index: the labels for the rows.
- dtype: the data type of a column, such as integer, decimal, text or date.
Setup
python -m venv .venv
source .venv/bin/activate
pip install pandas openpyxl matplotlib
openpyxl lets pandas read and write .xlsx files. In your code:
import pandas as pd
This course uses pandas 3.x. Differences from 2.x that matter for beginners are noted where they appear: text columns have their own str type, and changes to a derived table never alter the original (copy-on-write).
Make a small DataFrame
Throughout this course we reuse the same sample, which mirrors the SQL course:
orders = pd.DataFrame({
"order_id": [101, 102, 103, 104, 105, 106],
"customer_id": [1, 1, 2, 3, 3, 1],
"order_date": pd.to_datetime(["2025-09-01", "2025-09-15", "2025-09-20",
"2025-10-02", "2025-10-10", "2025-10-20"]),
"amount": [1200, 800, 1500, 600, 900, 500],
"status": ["paid", "paid", "refunded", "paid", "pending", "paid"],
})
A dictionary maps each column name to its list of values. In real work you will load a file instead.
Load a file
orders = pd.read_csv("orders.csv", parse_dates=["order_date"])
sheet = pd.read_excel("sales.xlsx", sheet_name="Oct")
Useful options of read_csv: sep=";" for another separator, encoding="utf-8" (or "latin-1" for old files), usecols=[...] to read only some columns, dtype={"pincode": "str"} to stop codes losing leading zeros, na_values=["NA", "-"] to mark extra missing-value symbols, and nrows=1000 to peek at a big file.
The five commands to run first
orders.head() # first 5 rows (use .head(10) for more; .tail() for the last)
orders.shape # (6, 5): rows, columns
orders.dtypes # the type of each column
orders.info() # columns, non-null counts, dtypes, memory
orders.describe() # count, mean, std, min, quartiles, max of numeric columns
Sample results:
>>> orders.shape
(6, 5)
>>> orders.dtypes
order_id int64
customer_id int64
order_date datetime64[us]
amount int64
status str
dtype: object
(The date shows datetime64[us] in pandas 3; older versions show [ns]. The text column shows str in 3.x and object in 2.x.) Read info() carefully: a number column that shows as text, or a date column that shows as text, is the first sign of dirty data. describe(include="all") also summarises text columns, and orders["status"].value_counts() counts each category:
status
paid 4
refunded 1
pending 1
Name: count, dtype: int64
Series: one column
orders["amount"] # a Series (one column)
orders["amount"].sum() # 5500
orders["amount"].mean() # 916.666...
orders["amount"].max() # 1500
orders["status"].unique() # the distinct values
orders["customer_id"].nunique() # how many distinct: 3
A Series has an index (the row labels, 0 to 5 here) and a dtype. Use two brackets for several columns, which returns a DataFrame:
orders[["order_id", "amount"]]
Create and change columns
New columns come from calculations on existing ones, applied to every row at once (no loop):
orders["amount_with_gst"] = orders["amount"] * 1.18
orders["is_paid"] = orders["status"] == "paid"
orders["month"] = orders["order_date"].dt.to_period("M")
orders["big_order"] = orders["amount"] >= 1000
.dt gives access to date parts (.dt.year, .dt.month, .dt.day_name()). Rename or drop:
orders = orders.rename(columns={"amount": "order_value"})
orders = orders.drop(columns=["is_paid"])
orders = orders.sort_values("order_value", ascending=False)
Notice the pattern: methods return a new DataFrame, and you assign it back. In pandas 3 this is the rule everywhere: derived tables are independent copies in behaviour (copy-on-write), so changing a subset never silently changes orders.
The index
Rows are identified by the index. The default is 0, 1, 2.... Make a column the index if it is a natural key:
by_id = orders.set_index("order_id")
by_id.loc[103] # the row for order 103
by_id = by_id.reset_index() # back to a normal column
After filtering or sorting the index labels stay with their rows (so you may see 3, 0, 5). Use reset_index(drop=True) for a fresh 0..n-1.
Save the result
orders.to_csv("orders_clean.csv", index=False) # index=False avoids an extra column
orders.to_excel("orders_clean.xlsx", index=False)
Common mistakes
- Not looking at the data first (
head,dtypes,info). - Dates and numbers read as text, because of stray symbols; fix with
parse_dates,dtypeor conversion lessons later. - Looping over rows with
forinstead of whole-column operations, which is slow and long. - Saving with the index, adding an unwanted
Unnamed: 0column on reload. - Forgetting to assign the result of a method such as
droporsort_values.
Try it yourself
- Install pandas and recreate the
ordersDataFrame. Printshape,dtypesanddescribe(). - Add a column for GST at 18 percent and a flag for orders over 1000.
- Count orders per status with
value_counts(). - Save the table to CSV without the index and read it back, checking that the dates are still dates (
parse_dates). - Load any CSV from your computer and write down three things
info()tells you about it.
Check yourself
What is the difference between a DataFrame and a Series?
A DataFrame is a two-dimensional table with several columns. A Series is a single column with its index.
How do you select two columns?
With a list inside brackets: df[["a", "b"]], which returns a DataFrame.
Why use index=False in to_csv?
Otherwise pandas writes the row index as an extra first column, which shows up as Unnamed: 0 when you read the file again.
Which commands do you run first on a new dataset?
head(), shape, dtypes, info() and describe(), plus value_counts() on categories.
Interview questions
How do you add a calculated column in pandas?
Assign a calculation on existing columns to a new column name, such as df["total"] = df["price"] * df["qty"]; it is applied to all rows at once (vectorised).
Why avoid looping over DataFrame rows?
Vectorised operations run in optimised compiled code and are much faster and shorter than Python loops; iterrows is slow and rarely needed.
What does copy-on-write mean in pandas 3?
Any DataFrame or Series derived from another behaves as an independent copy, so modifying it never changes the original, and pandas copies data lazily only when needed.
Summary
- A DataFrame is a table; a Series is one column; the index labels the rows.
- Load with
read_csvorread_excel, then inspect withhead,shape,dtypes,info,describe. - Create columns with whole-column arithmetic; methods return new objects, so assign them back.
- Save with
to_csv(index=False).
Going deeper
- 10 minutes to pandas: the official quick tour.
- Intro to data structures: Series and DataFrame in detail.
- IO tools: every
read_*andto_*option. - Copy-on-Write: the pandas 3 behaviour.
See how much of this you can explain
A free mock interview asks you new questions on Python for data analysis with pandas for Data Analyst and scores every answer, with what to fix.