Python – Working with Data

This tutorial provides a step-by-step guide to working with data in Python, covering essential concepts like data structures, file handling, data manipulation, and visualization. Each step builds on the previous one, with practical examples and engaging exercises to reinforce learning.

Beginner30 minutes

Step 1: Step 1: Understanding Basic Data Structures

Python provides several built-in data structures like lists, tuples, dictionaries, and sets. These are essential for storing and organizing data efficiently.

- Lists: Ordered, mutable collections.

- Tuples: Ordered, immutable collections.

- Dictionaries: Key-value pairs for quick lookups.

- Sets: Unordered collections of unique elements.

# Example of basic data structures
my_list = [1, 2, 3, 4]
my_tuple = (1, 2, 3)
my_dict = {'name': 'Alice', 'age': 25}
my_set = {1, 2, 3, 4}

Practice Exercise

Create a dictionary to store information about a book (title, author, year published). Then, extract and print the author's name.

Show Solution

# Solution
book = {'title': '1984', 'author': 'George Orwell', 'year': 1949}
print(book['author'])  # Output: George Orwell

Step 2: Step 2: Reading and Writing Files

Python makes it easy to work with files. You can read data from files (e.g., CSV, text) and write data back to them.

- Use open() to open a file.

- Use read() or readlines() to read content.

- Use write() to write content.

# Example: Reading and writing a text file
with open('example.txt', 'r') as file:
    content = file.read()

with open('output.txt', 'w') as file:
    file.write('Hello, World!')

Practice Exercise

Create a text file named notes.txt and write three lines of text into it. Then, read the file and print the second line.

Show Solution

# Solution
with open('notes.txt', 'w') as file:
    file.write('Line 1\nLine 2\nLine 3')

with open('notes.txt', 'r') as file:
    lines = file.readlines()
    print(lines[1].strip())  # Output: Line 2

Step 3: Step 3: Working with CSV Data

CSV (Comma-Separated Values) files are commonly used for storing tabular data. Python's csv module simplifies reading and writing CSV files.

- Use csv.reader() to read CSV data.

- Use csv.writer() to write CSV data.

import csv

# Example: Reading a CSV file
with open('data.csv', 'r') as file:
    reader = csv.reader(file)
    for row in reader:
        print(row)

Practice Exercise

Given a CSV file students.csv with columns Name, Age, and Grade, calculate and print the average age of the students.

Show Solution

# Solution
import csv

ages = []
with open('students.csv', 'r') as file:
    reader = csv.DictReader(file)
    for row in reader:
        ages.append(int(row['Age']))

average_age = sum(ages) / len(ages)
print(f'Average Age: {average_age}')

Step 4: Step 4: Data Manipulation with Pandas

Pandas is a powerful library for data manipulation and analysis. It introduces two key data structures: Series (1D) and DataFrame (2D).

- Use pd.read_csv() to load data.

- Use df.head() to view the first few rows.

- Use df.describe() to get summary statistics.

import pandas as pd

# Example: Loading and exploring data
df = pd.read_csv('data.csv')
print(df.head())
print(df.describe())

Practice Exercise

Load a CSV file sales.csv with columns Product, Quantity, and Price. Calculate the total revenue for each product and display the results.

Show Solution

# Solution
import pandas as pd

df = pd.read_csv('sales.csv')
df['Revenue'] = df['Quantity'] * df['Price']
total_revenue = df.groupby('Product')['Revenue'].sum()
print(total_revenue)

Step 5: Step 5: Data Visualization with Matplotlib

Matplotlib is a popular library for creating visualizations. You can create line plots, bar charts, scatter plots, and more.

- Use plt.plot() for line plots.

- Use plt.bar() for bar charts.

- Use plt.scatter() for scatter plots.

import matplotlib.pyplot as plt

# Example: Creating a line plot
x = [1, 2, 3, 4]
y = [10, 20, 25, 30]
plt.plot(x, y)
plt.xlabel('X-axis')
plt.ylabel('Y-axis')
plt.title('Simple Line Plot')
plt.show()

Practice Exercise

Create a bar chart showing the total revenue for each product from the sales.csv file used in Step 4.

Show Solution

# Solution
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('sales.csv')
df['Revenue'] = df['Quantity'] * df['Price']
total_revenue = df.groupby('Product')['Revenue'].sum()

plt.bar(total_revenue.index, total_revenue.values)
plt.xlabel('Product')
plt.ylabel('Revenue')
plt.title('Revenue by Product')
plt.show()

Step 6: Step 6: Cleaning and Preprocessing Data

Real-world data is often messy. Pandas provides tools for cleaning and preprocessing data, such as handling missing values, removing duplicates, and transforming data.

- Use df.dropna() to remove missing values.

- Use df.drop_duplicates() to remove duplicates.

- Use df.fillna() to fill missing values.

# Example: Cleaning data
df = pd.read_csv('data.csv')
df.dropna(inplace=True)  # Remove rows with missing values
df.drop_duplicates(inplace=True)  # Remove duplicate rows

Practice Exercise

Given a dataset employees.csv with missing values in the Salary column, replace the missing values with the median salary.

Show Solution

# Solution
import pandas as pd

df = pd.read_csv('employees.csv')
median_salary = df['Salary'].median()
df['Salary'].fillna(median_salary, inplace=True)
print(df)

Step 7: Step 7: Combining Data with Pandas

Pandas allows you to combine datasets using operations like concatenation, merging, and joining.

- Use pd.concat() to concatenate DataFrames.

- Use pd.merge() to merge DataFrames based on a key.

# Example: Merging two DataFrames
df1 = pd.DataFrame({'ID': [1, 2], 'Name': ['Alice', 'Bob']})
df2 = pd.DataFrame({'ID': [1, 2], 'Age': [25, 30]})
df_merged = pd.merge(df1, df2, on='ID')
print(df_merged)

Practice Exercise

Given two CSV files customers.csv and orders.csv, merge them on the CustomerID column and display the combined data.

Show Solution

# Solution
import pandas as pd

customers = pd.read_csv('customers.csv')
orders = pd.read_csv('orders.csv')
combined = pd.merge(customers, orders, on='CustomerID')
print(combined)

Step 8: Step 8: Advanced Data Analysis with GroupBy

The groupby function in Pandas allows you to group data based on one or more columns and perform aggregate operations like sum, mean, or count.

- Use df.groupby() to group data.

- Use aggregation functions like sum(), mean(), or count().

# Example: Grouping and aggregating data
df = pd.read_csv('sales.csv')
grouped = df.groupby('Product')['Quantity'].sum()
print(grouped)

Practice Exercise

Using the sales.csv file, group the data by Product and calculate the average price for each product.

Show Solution

# Solution
import pandas as pd

df = pd.read_csv('sales.csv')
average_price = df.groupby('Product')['Price'].mean()
print(average_price)

Step 9: Step 9: Exporting Data to Different Formats

Pandas allows you to export data to various formats like CSV, Excel, and JSON.

- Use df.to_csv() to export to CSV.

- Use df.to_excel() to export to Excel.

- Use df.to_json() to export to JSON.

# Example: Exporting data to CSV
df = pd.read_csv('data.csv')
df.to_csv('exported_data.csv', index=False)

Practice Exercise

Load the sales.csv file, calculate the total revenue for each product, and export the results to a new CSV file named revenue_summary.csv.

Show Solution

# Solution
import pandas as pd

df = pd.read_csv('sales.csv')
df['Revenue'] = df['Quantity'] * df['Price']
revenue_summary = df.groupby('Product')['Revenue'].sum().reset_index()
revenue_summary.to_csv('revenue_summary.csv', index=False)

Step 10: Step 10: Real-World Data Project

Combine all the skills learned in this tutorial to analyze a real-world dataset. For example, analyze sales data to identify trends, clean the data, and create visualizations.

# Example: Real-world data analysis
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('sales_data.csv')
df['Revenue'] = df['Quantity'] * df['Price']

# Clean data
df.dropna(inplace=True)

# Analyze trends
trends = df.groupby('Month')['Revenue'].sum()

# Visualize trends
plt.plot(trends.index, trends.values)
plt.xlabel('Month')
plt.ylabel('Revenue')
plt.title('Monthly Revenue Trends')
plt.show()

Practice Exercise

Find a dataset online (e.g., from Kaggle or a government open data portal). Clean the data, perform an analysis, and create at least two visualizations to present your findings.

Show Solution

# Solution will vary based on the dataset chosen.
# Example structure:
import pandas as pd
import matplotlib.pyplot as plt

# Load and clean data
df = pd.read_csv('your_dataset.csv')
df.dropna(inplace=True)

# Perform analysis
analysis_result = df.groupby('Category')['Value'].mean()

# Create visualizations
plt.bar(analysis_result.index, analysis_result.values)
plt.xlabel('Category')
plt.ylabel('Average Value')
plt.title('Average Value by Category')
plt.show()

# Second visualization (e.g., line plot)
plt.plot(df['Date'], df['Value'])
plt.xlabel('Date')
plt.ylabel('Value')
plt.title('Value Over Time')
plt.show()