Solutions: Data Analysis with Python

Interactive, in-browser edition: completed code you can run and edit.

Author

Dr. Chester Ismay

NoteAbout this page

These are the completed solutions, runnable in your browser, no install needed. Click Run Code on any cell (run the import and data-loading cells near the top first). Edit any cell to experiment.

Looking for the blanks to fill in yourself? See the exercises page.

Intro: Foundations of Data Analysis with Python

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart LR
  F[("A CSV or<br/>Excel file")]:::data --> M1["Module 1<br/>Wrangle it<br/>with pandas"]:::proc
  M1 --> M2["Module 2<br/>Chart it<br/>matplotlib, seaborn"]:::proc
  M2 --> M3["Module 3<br/>Make it interactive<br/>plotly"]:::proc
  M3 --> M4["Module 4<br/>Tell the story"]:::accent
  M4 --> O[/"A finding someone<br/>can act on"/]:::good
  classDef data fill:#d1f0ec,stroke:#0b7a75,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef accent fill:#ede0ff,stroke:#6a1b9a,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a

The arc of the course: a file becomes a table, the table becomes a chart, and the chart becomes something someone can act on.

Walkthrough #1: Setting Up the Python Environment

If you haven’t already installed Python, Jupyter, and the necessary packages, there are instructions on the course repo in the README to do so here.

Exercise #1: Setting Up the Python Environment

By completing this exercise, you will be able to - Import necessary Python packages - Check for successful package loading

Follow the instructions above in Walkthrough to check for correct installation of necessary packages. We’ll wait a few minutes to make sure as many of you are set up as possible. Please give a thumbs up in the pulse check if you are ready to move on.

Module 1: Data Wrangling with Pandas

Walkthrough #2: Loading and Inspecting Data with Pandas

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  C[("economies.csv")]:::data --> RC["read_csv()"]:::proc
  X[("economies.xlsx")]:::data --> RX["read_excel()<br/>needs openpyxl"]:::proc
  RC --> DF[/"One DataFrame"/]:::good
  RX --> DF
  DF --> Q1["head()<br/>what does a row<br/>look like?"]:::accent
  DF --> Q2["info()<br/>what type is each<br/>column, and what<br/>is missing?"]:::accent
  DF --> Q3["describe()<br/>what range are<br/>the numbers in?"]:::accent
  classDef data fill:#d1f0ec,stroke:#0b7a75,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a
  classDef accent fill:#ede0ff,stroke:#6a1b9a,color:#1a1a1a

Either file type lands in the same DataFrame. Three calls then answer three questions about it.

Import data from a CSV or from an Excel file

Common Pitfall: read_excel relies on the openpyxl engine for .xlsx files. If you see an ImportError or Missing optional dependency 'openpyxl', install it with pip install openpyxl. Also note that read_excel reads only the first sheet by default; pass sheet_name= if your workbook has multiple sheets.

Perform an initial exploration of the data

Exercise #2: Loading and Inspecting Data with Pandas

By completing this exercise, you will be able to use pandas to - Import data from a CSV or from an Excel file - Perform an initial exploration of the data

Walkthrough #3: Cleaning and Preparing Data with Pandas

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart LR
  R[("Raw<br/>DataFrame")]:::data --> A["1. Missing values<br/>dropna, fillna"]:::proc
  A --> B["2. Types<br/>astype"]:::proc
  B --> C["3. Names<br/>rename"]:::proc
  C --> E["4. Rows you want<br/>boolean filter"]:::proc
  E --> T[/"A table you can<br/>analyze"/]:::good
  classDef data fill:#d1f0ec,stroke:#0b7a75,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a

Cleaning has an order. Settle what is missing, then types, then names, and cut down to the rows you want last.

Handle missing data

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  M{"A column has<br/>missing values"}:::decision
  M -->|"a few rows, and you<br/>can spare them"| DR["dropna(subset=[...])<br/>drop those rows"]:::proc
  M -->|"you need to keep<br/>every row"| FI["fillna(value)<br/>put something in<br/>the gap"]:::accent
  FI --> W["A filled value is a<br/>guess. Say so when<br/>you report the number."]:::warn
  classDef decision fill:#fff3cd,stroke:#d39e00,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef accent fill:#ede0ff,stroke:#6a1b9a,color:#1a1a1a
  classDef warn fill:#ffe8cc,stroke:#e8730c,color:#1a1a1a

Two ways to handle a gap. Which one is right depends on whether you can spare the rows.

Remove rows

Replace missing values with specific value

This can be extended to replace missing values with the mean, median, or mode of the column too.

Common Pitfall: Methods like dropna, fillna, and rename return a new DataFrame by default and do NOT modify the original in place. Assign the result to a variable (or pass inplace=True) or your changes will be lost. Reassigning, as done here, is the recommended pattern.

Convert a column to a different data type

Rename a column

Filtering rows based on conditions

Conditions on a single column

Common Pitfall: When combining multiple conditions, wrap each comparison in parentheses and use & (and) / | (or), not the Python keywords and/or. Forgetting the parentheses, e.g. economies['inflation_rate'] < 0 & ..., triggers an operator-precedence error because & binds more tightly than <.

Conditions on multiple columns

Exercise #3: Cleaning and Preparing Data with Pandas

By completing this exercise, you will be able to use pandas to - Handle missing data - Convert a column to a different data type - Rename a column - Filter a DataFrame

Handle Missing Data

Remove rows

Replace missing values with specific value

Convert a Column to a Different Data Type and Rename a Column

Convert a Column to a Different Data Type

Rename a Column

Filter a DataFrame

Walkthrough #4: Transforming and Aggregating Data with Pandas

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart LR
  T[("One row per<br/>country and year")]:::data --> S["SPLIT<br/>groupby('income_group')"]:::proc
  S --> A["APPLY<br/>mean(), agg({...})"]:::proc
  A --> C["COMBINE"]:::proc
  C --> O[/"One row per<br/>income group"/]:::good
  classDef data fill:#d1f0ec,stroke:#0b7a75,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a

Split, apply, combine: groupby splits the table, the aggregation runs once per group, and pandas stitches the answers back together.

Grouping data

Applying Functions

Applying a function element-wise with map()

Applying a Function to Groups with groupby() and agg()

Summary tables

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  Q{"What goes in<br/>the cells?"}:::decision
  Q -->|"counts, crossing<br/>two categories"| CT["pd.crosstab(a, b)"]:::proc
  Q -->|"an aggregate of a<br/>numeric column"| PT["pd.pivot_table(values=,<br/>index=, columns=, aggfunc=)"]:::proc
  Q -->|"counts within<br/>one category"| VC["series.value_counts()"]:::proc
  classDef decision fill:#fff3cd,stroke:#d39e00,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a

Three summary tools, one question: what do you want sitting in the cells?

Analyzing categorical data

Using cross-tabulation

By getting group counts

Exercise #4: Transforming and Aggregating Data with Pandas

By completing this exercise, you will be able to use pandas to - Aggregate data effectively by grouping it - Transform data by applying functions element-wise or to groups - Create summary tables - Analyze categorical data using cross-tabulation and counts

Grouping Data

Applying Functions

Applying a function element-wise with map()

Applying a function to groups with groupby() and agg()

Summary Tables

Analyzing Categorical Data

Using Cross-Tabulation

By Getting Group Counts

Self-Check

By the end of this module, you should be able to:

Module 2: Data Visualization Basics with Matplotlib and Seaborn

Walkthrough #5: Data Visualization Techniques with Matplotlib and Seaborn

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart LR
  F["plt.figure()<br/>start a NEW canvas"]:::key --> P["plt.plot(), plt.bar()<br/>draw the data"]:::proc
  P --> L["xlabel, ylabel, title<br/>say what it shows"]:::proc
  L --> A["ylim, xticks, grid<br/>tidy the axes"]:::proc
  A --> S["plt.show()<br/>render it"]:::good
  F -.->|"skip this one"| X["and the next chart<br/>draws on top of this one"]:::bad
  classDef key fill:#fff3cd,stroke:#d39e00,stroke-width:2px,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a
  classDef bad fill:#fde0e0,stroke:#c62828,color:#1a1a1a

Every matplotlib chart follows the same five steps, and the first one is the step people skip.

Line plot

Bar chart

Adding labels and titles

Adjusting axes and tick marks

Common Pitfall: Call plt.figure() before each new plot and plt.show() after it. If you skip plt.figure(), matplotlib keeps drawing onto the previous figure, and successive plots can overlap or accumulate unexpectedly within the same axes.

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  Q{"What are you<br/>trying to see?"}:::decision
  Q -->|"how one number<br/>is spread out"| H["histplot<br/>bins plus a density curve"]:::proc
  Q -->|"the summary of one number<br/>inside each category"| B["boxplot<br/>median, quartiles, outliers"]:::proc
  Q -->|"the full shape of one number<br/>inside each category"| V["violinplot<br/>the whole distribution"]:::proc
  classDef decision fill:#fff3cd,stroke:#d39e00,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a

Pick the seaborn plot from the question you are asking, not from the plot you feel like making.

Histogram with Seaborn

Boxplot with Seaborn

Violin plot with Seaborn

Exercise #5: Data Visualization Techniques with Matplotlib and Seaborn

By completing this exercise, you will be able to use matplotlib and seaborn to - Create line plots and bar charts - Add labels and titles - Adjust axes and tick marks - Draw a histogram, boxplot, and violin plot

Line Plot

Bar Chart

Adding Labels and Titles

Adjusting Axes and Tick Marks

Histogram with Seaborn

Boxplot with Seaborn

Violin Plot with Seaborn

Self-Check

By the end of this module, you should be able to:

Module 3: Interactive Data Visualization with Plotly

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart TD
  D[("The same<br/>DataFrame")]:::data --> MPL["matplotlib, seaborn<br/>a fixed picture"]:::proc
  D --> PLY["plotly<br/>hover, zoom, select"]:::accent
  PLY --> PX["plotly.express<br/>one line from<br/>a tidy frame"]:::proc
  PLY --> GO["graph_objects with<br/>make_subplots<br/>one panel at a time"]:::proc
  PX --> SH["fig.show()"]:::key
  GO --> SH
  SH --> O[/"An interactive<br/>figure"/]:::good
  classDef data fill:#d1f0ec,stroke:#0b7a75,color:#1a1a1a
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef accent fill:#ede0ff,stroke:#6a1b9a,color:#1a1a1a
  classDef key fill:#fff3cd,stroke:#d39e00,stroke-width:2px,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a

Plotly has two entry points: express for a quick chart from a tidy frame, graph_objects when you are assembling panels. Both end at fig.show().

Walkthrough #6: Interactive Charts and Dashboards with Plotly

Basic interactive chart

Common Pitfall: If a Plotly figure shows up blank in a Jupyter notebook, the renderer was likely not initialized. Make sure the setup cell that runs pyo.init_notebook_mode(connected=True) was executed first, and remember that you must call fig.show(), simply creating fig does not display anything.

Adding interactive elements

Designing a simple dashboard

Common Pitfall: Plotly subplot positions are 1-indexed, so the first cell is row=1, col=1 (not 0). The rows/cols you pass to make_subplots must also be large enough to hold every add_trace position, or you will get an index error.

Exercise #6: Interactive Charts and Dashboards with Plotly

By completing this exercise, you will be able to use plotly to - Create a basic interactive chart - Add interactive elements: hover, zoom, and selection tools - Design a simple dashboard with multiple charts

Basic Interactive Chart

Adding Interactive Elements

Designing a Simple Dashboard

Self-Check

By the end of this module, you should be able to:

Module 4: Real-World Data Analysis Project

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'wrappingWidth': 500}}}%%
flowchart LR
  P["Pick a dataset<br/>and a question"]:::proc --> C["Clean and<br/>transform"]:::proc
  C --> A["Analyze:<br/>group, chart,<br/>compare"]:::proc
  A --> F["Write the<br/>finding down"]:::accent
  F --> FB{"Does it answer<br/>the question?"}:::decision
  FB -->|"not yet"| C
  FB -->|"yes"| PR[/"Visuals plus<br/>narrative"/]:::good
  classDef proc fill:#dbe9ff,stroke:#1a73e8,color:#1a1a1a
  classDef accent fill:#ede0ff,stroke:#6a1b9a,color:#1a1a1a
  classDef decision fill:#fff3cd,stroke:#d39e00,color:#1a1a1a
  classDef good fill:#d8f5d8,stroke:#2e7d32,color:#1a1a1a

The project is a loop, not a line. Expect to go back to cleaning at least once.

Walkthrough: Applying Your Skills, From Data to Insights

Selecting a Dataset

Questions to Ask:

  1. What industry problem or area of interest does the dataset align with?
    • Is the dataset relevant to economic analysis, market research, policy planning, or another industry?
  2. Does the dataset provide sufficient complexity and scope for a thorough analysis?
    • Does it include multiple variables and data points across different time periods and categories (e.g., income groups, countries)?
  3. What specific questions or hypotheses do we want to explore with this dataset?
    • Are we interested in comparing economic indicators across countries, understanding the impact of GDP per capita on other variables, or identifying trends over time?

Example:

  • Dataset: The economies dataset.
  • Industry Problem: Understanding economic disparities between countries and the impact of economic indicators on overall economic health.
  • Specific Questions:
    • How do GDP per capita and gross savings vary across different income groups?
    • How has the inflation rate changed over time for specified income groups?

Applying Cleaning, Transforming, and Analysis Techniques

Questions to Ask:

  1. What cleaning steps are necessary to prepare the data for analysis?
    • Are there any missing values that need to be handled? Are there any inconsistencies in data types?
  2. What transformations are required to make the data analysis-ready?
    • Do we need to create new columns, filter specific rows, or aggregate data by certain categories?
  3. How can we analyze the data to uncover patterns, trends, or anomalies?
    • What statistical methods or visualizations can we use to explore relationships between variables?

Example:

  • Cleaning:

Common Pitfall: Build new columns on a DataFrame you fully own (here economies_cleaned, created by fillna), not on a filtered slice of another DataFrame. Since pandas 3, copy-on-write is the default: assigning to a column of a slice no longer warns (SettingWithCopyWarning is gone) and never updates the original, and chained assignment raises a ChainedAssignmentError instead of silently half-working. When you need an independent copy, use .copy().

  • Transforming:

  • Analyzing:

Initial Findings and Interpretation

Questions to Ask:

  1. What do the initial findings tell us about the data?
    • Are there any notable patterns or trends in the data? Are there any unexpected results?
  2. How do these insights relate to the problem defined earlier?
    • Do the findings help us understand economic disparities between countries? Do they provide insights into the impact of certain economic indicators?
  3. What hypotheses can we test based on the initial results?
    • Can we test hypotheses about the relationship between GDP per capita and other economic indicators? Can we refine our analysis to explore these hypotheses further?

Example:

  • Initial Findings:
    • GDP per Capita vs. Gross Savings: The scatter plot shows that high-income countries generally have higher GDP per capita and gross savings. There seems to be a slight positive correlation between these two indicators.
    • Inflation Rate Over Time: The line plot indicates that inflation rates vary significantly over time and across different income groups. Low and lower middle income countries tend to experience higher volatility in inflation rates.
  • Interpretation:
    • These findings suggest that economic health, as measured by GDP per capita and gross savings, is strongly influenced by the income group of a country. High-income countries appear to have more stable and higher economic performance.
    • The volatility in inflation rates among low-income countries may indicate economic instability, which could be a key area for policy intervention.
  • Hypotheses:
    • Hypothesis 1: High-income countries have a higher average GDP per capita and gross savings compared to low-income countries.
    • Hypothesis 2: Low-income countries experience greater volatility in inflation rates compared to high-income countries.
  • Next Steps:
    • Conduct further analysis to test these hypotheses, using statistical methods to confirm the observed patterns.
    • Explore other economic indicators to gain a more comprehensive understanding of economic disparities and trends.

By following these steps, you can effectively select, clean, transform, and analyze the economies dataset to gain valuable insights and address common industry problems or research questions.

Walkthrough: Finalizing and Presenting Your Data Analysis Project

Integrate Feedback to Refine the Analysis

Questions to Ask:

  1. What feedback have you received from peers, stakeholders, or mentors?
    • Is there feedback on the clarity of the analysis, choice of visualizations, or the comprehensiveness of the analysis?
  2. How can you incorporate this feedback into your analysis?
    • Are there additional variables that need to be analyzed? Do you need to clean the data further or adjust the visualizations?
  3. What new questions or hypotheses have emerged from the feedback?
    • Does the feedback suggest new directions for the analysis or areas that need more focus?

Example:

  • Feedback:
    • Peers suggested that the analysis should also consider the impact of unemployment rates.
    • Stakeholders requested more clarity on the relationship between GDP per capita and inflation rates across different income groups.
  • Refining the Analysis:
    • Additional data needs to be found to meet the request for more clarity. Or maybe a further drilldown on specific countries would be helpful?

Finalize the Presentation with Impactful Visuals and Narrative

Questions to Ask:

  1. What are the key insights from the analysis that need to be highlighted?
    • What are the most important findings that should be communicated to the audience?
  2. How can you create impactful visuals that clearly convey these insights?
    • What types of charts or visualizations best represent the data and findings?
  3. What narrative will you use to guide the audience through the presentation?
    • How will you structure the presentation to tell a compelling story with the data?

Example:

  • Key Insights:

    • High-income countries have higher GDP per capita and gross savings.
    • There is a positive correlation between GDP per capita and gross savings.
    • Low-income countries experience greater volatility in inflation rates.
    • Unemployment rates vary significantly across income groups.

  • Impactful Visuals:

  • Narrative:

    • Introduction: Introduce the dataset and the industry problem. Explain why understanding economic indicators across different income groups is important.
    • Key Findings: Present the key findings using the visualizations created. Highlight the relationship between GDP per capita, gross savings, inflation rates, and unemployment rates.
    • Detailed Analysis: Dive deeper into each key finding, providing more context and interpretation. Explain the significance of the trends and patterns observed in the data.
    • Conclusion: Summarize the insights and discuss potential implications for policy or business decisions. Suggest areas for further research or analysis based on the findings.

Rehearse the Presentation

Questions to Ask:

  1. How will you structure your presentation to ensure a smooth flow?
    • What order will you present the visualizations and insights? How will you transition between different sections?
  2. How will you engage your audience and ensure they understand the key points?
    • What techniques will you use to highlight important information and keep the audience’s attention?
  3. What potential questions or feedback might you receive, and how will you address them?
    • How will you prepare for questions about the data, analysis methods, or findings?

Example:

  • Structuring the Presentation:
    • Start with an overview of the dataset and the industry problem.
    • Move on to the key findings, using the most impactful visualizations to illustrate each point.
    • Provide a detailed analysis of each finding, explaining the significance and implications.
    • Conclude with a summary of insights and suggestions for further research.
  • Engaging the Audience:
    • Use clear and concise language to explain complex concepts.
    • Highlight key points using annotations or callouts on the visualizations.
    • Encourage questions and interaction to keep the audience engaged.
  • Preparing for Questions:
    • Anticipate common questions about the data sources, cleaning methods, and analysis techniques.
    • Prepare explanations for any limitations of the data or analysis.
    • Be ready to discuss potential next steps and areas for further research based on the findings.

By following these steps, you can effectively integrate feedback, finalize your presentation with impactful visuals and narrative, and rehearse to ensure a smooth and engaging delivery.

Self-Check

By the end of this module, you should be able to: