Student Setup and Lecture Explorer

Choose how far you want to go: view, explore, reproduce, or extend

Explore with Lecture Explorer

This repository includes a project-specific agent skill that helps you navigate the lecture, configure a fork, check data access, and plan extensions. With pi, review AGENTS.md and .agents/skills/lecture-explorer/SKILL.md, then approve the project-trust prompt so pi can load the project-local skill. Run:

/skill:lecture-explorer Where should I start?

For a more ambitious workflow:

/skill:lecture-explorer Outline the steps to allow me to extend this project on my own machine.

Lecture Explorer is a repository guide, not a tutor or grader. It explains what is available, points to the relevant files, and shows commands before making changes.

Level 1: View the archived lecture

No local setup, data, or credentials are required.

The committed files under docs/ are the archived rendered outputs.

Level 2: Fork and explore the repository

You need a GitHub account and Git. GitHub CLI (gh) is optional.

Fork in a browser

Open https://github.com/CasparDP/phd-lecture-dec.
Select Fork and create the fork under your account.

Clone your fork:

git clone https://github.com/YOUR-USERNAME/phd-lecture-dec.git
cd phd-lecture-dec

Confirm that the instructor repository is available as upstream:
```
git remote -v
git remote add upstream https://github.com/CasparDP/phd-lecture-dec.git
```
Run the second command only if an upstream remote is not already present.
Work on a branch in your fork:
```
git switch -c lecture-exploration
```

Where to start

index.qmd: lecture landing page and learning goals
lecture-materials/slides/lecture-slides.qmd: complete lecture narrative
lecture-materials/handouts/institutional-background.qmd: USITC institutional background
lecture-materials/playground/revisiting-jones.qmd: Jones analysis and research design
lecture-materials/playground/website-data.qmd: exploratory website-text analysis
paper_summaries/: AI-generated paper summaries that should be checked against the papers
data-prep/Scripts/README.md: data-pipeline overview

Level 3: Reproduce or extend an analysis

A full rebuild is not required to use the lecture. The input sources differ in how they are obtained, but none of the generated CSV or DuckDB files needs to be committed to Git.

Software

Component	Purpose	Notes
Quarto	Render the website and RevealJS slides	Rendering the complete project may execute analysis pages.
R and R packages	Data preparation, regressions, tables, and figures	`data-prep/Scripts/load_libraries.R` installs missing packages, so review it before running.
Python 3.13	USITC collection and industry matching	Use Python 3.13. The current Torch dependency does not provide a compatible Python 3.14 wheel.
Poetry	Python environment and commands	Use `poetry install` and `poetry run ...`.
Ollama with `gpt-oss:20b`	LLM-assisted SIC/NAICS matching	The model name is currently hard-coded in `industry_match_chat.py`.
DuckDB	Local intermediate tables	The scripts create `data-prep/DB/jones_duckdb`; generated databases are ignored by Git.

Data availability

Input	How to obtain it
Lecture source, rendered pages, figures, and paper summaries	Included in this repository
USITC investigation data	Regenerated from the public USITC website with `get_data.py` and `select_cases.R`
SIC/NAICS crosswalk	Included in this repository or refreshed with `naics_sic_xwalk_DL.R`
Compustat firm and accounting data	Downloaded by `jones.R` with a WRDS account that has Compustat access
Jones residuals and analysis DuckDB tables	Generated locally by the included R scripts
Cleaned historical website text	Download `TXT_combined.zip` from the open-access CompuCrawl database

A fresh run can rebuild the local DuckDB. Exact archived estimates may nevertheless differ if USITC or Compustat data have changed or if the local LLM produces different industry matches.

Configure the website-text path

Download and extract TXT_combined.zip from the CompuCrawl data page. Then create your local configuration:

cp config_example.R config.R

Edit config.R and set:

WEBSITE_TEXT_DIR <- "/absolute/path/to/TXT_combined"

config.R is ignored by Git. Each user can therefore keep data wherever convenient without changing the shared analysis source.

WRDS credentials

The Jones data-preparation script reads WRDS credentials from a local secrets.csv:

cp secrets-example.csv secrets.csv

Fill in secrets.csv locally. It is excluded by .gitignore.

Never commit secrets.csv.
Never paste credentials into an AI-agent conversation.
If WRDS authentication fails, stop. Do not retry automatically.

Pipeline order

Run commands from the repository root and validate each output before continuing:

Install the Python environment:
```
poetry install
```

Collect and clean public USITC data:

poetry run python data-prep/Scripts/get_data.py
Rscript data-prep/Scripts/select_cases.R

Match investigations to industries with the local Ollama model:
```
poetry run python data-prep/Scripts/industry_match_chat.py
```
Download Compustat data and estimate Jones residuals:
```
Rscript data-prep/Scripts/jones.R
```

Build cohorts and analysis tables:

Rscript data-prep/Scripts/phd-lecture-analyses.R

Render only the page you intend to inspect after validating its inputs.

The scripts use data-prep/DB/jones_duckdb as the canonical database. They create the local database and its tables during the pipeline.

Safe starting checks

These commands inspect your setup but do not install software or execute analyses:

git status --short --branch
git remote -v
command -v git
command -v quarto
command -v R
command -v Rscript
command -v poetry
command -v ollama

After the checks, choose one concrete target: inspect the lecture, render only the slides, rebuild the USITC stage, rebuild the Jones stage, or adapt an analysis in your fork.

--- title: "Student Setup and Lecture Explorer" subtitle: "Choose how far you want to go: view, explore, reproduce, or extend" format: html --- ## Explore with Lecture Explorer This repository includes a project-specific agent skill that helps you navigate the lecture, configure a fork, check data access, and plan extensions. With [pi](https://pi.dev), review `AGENTS.md` and `.agents/skills/lecture-explorer/SKILL.md`, then approve the project-trust prompt so pi can load the project-local skill. Run: ```text /skill:lecture-explorer Where should I start? ``` For a more ambitious workflow: ```text /skill:lecture-explorer Outline the steps to allow me to extend this project on my own machine. ``` Lecture Explorer is a repository guide, not a tutor or grader. It explains what is available, points to the relevant files, and shows commands before making changes. ## Level 1: View the archived lecture No local setup, data, or credentials are required. - [Lecture website](https://caspardp.github.io/phd-lecture-dec/) - [Lecture slides](https://caspardp.github.io/phd-lecture-dec/lecture-materials/slides/lecture-slides.html) - [Jones analysis](https://caspardp.github.io/phd-lecture-dec/lecture-materials/playground/revisiting-jones.html) - [Website analysis](https://caspardp.github.io/phd-lecture-dec/lecture-materials/playground/website-data.html) The committed files under `docs/` are the archived rendered outputs. ## Level 2: Fork and explore the repository You need a GitHub account and Git. GitHub CLI (`gh`) is optional. ### Fork in a browser 1. Open <https://github.com/CasparDP/phd-lecture-dec>. 2. Select **Fork** and create the fork under your account. 3. Clone your fork: ```bash git clone https://github.com/YOUR-USERNAME/phd-lecture-dec.git cd phd-lecture-dec ``` 4. Confirm that the instructor repository is available as `upstream`: ```bash git remote -v git remote add upstream https://github.com/CasparDP/phd-lecture-dec.git ``` Run the second command only if an `upstream` remote is not already present. 5. Work on a branch in your fork: ```bash git switch -c lecture-exploration ``` ### Where to start - `index.qmd`: lecture landing page and learning goals - `lecture-materials/slides/lecture-slides.qmd`: complete lecture narrative - `lecture-materials/handouts/institutional-background.qmd`: USITC institutional background - `lecture-materials/playground/revisiting-jones.qmd`: Jones analysis and research design - `lecture-materials/playground/website-data.qmd`: exploratory website-text analysis - `paper_summaries/`: AI-generated paper summaries that should be checked against the papers - `data-prep/Scripts/README.md`: data-pipeline overview ## Level 3: Reproduce or extend an analysis A full rebuild is not required to use the lecture. The input sources differ in how they are obtained, but none of the generated CSV or DuckDB files needs to be committed to Git. ### Software | Component | Purpose | Notes | |---|---|---| | Quarto | Render the website and RevealJS slides | Rendering the complete project may execute analysis pages. | | R and R packages | Data preparation, regressions, tables, and figures | `data-prep/Scripts/load_libraries.R` installs missing packages, so review it before running. | | Python 3.13 | USITC collection and industry matching | Use Python 3.13. The current Torch dependency does not provide a compatible Python 3.14 wheel. | | Poetry | Python environment and commands | Use `poetry install` and `poetry run ...`. | | Ollama with `gpt-oss:20b` | LLM-assisted SIC/NAICS matching | The model name is currently hard-coded in `industry_match_chat.py`. | | DuckDB | Local intermediate tables | The scripts create `data-prep/DB/jones_duckdb`; generated databases are ignored by Git. | ### Data availability | Input | How to obtain it | |---|---| | Lecture source, rendered pages, figures, and paper summaries | Included in this repository | | USITC investigation data | Regenerated from the public USITC website with `get_data.py` and `select_cases.R` | | SIC/NAICS crosswalk | Included in this repository or refreshed with `naics_sic_xwalk_DL.R` | | Compustat firm and accounting data | Downloaded by `jones.R` with a WRDS account that has Compustat access | | Jones residuals and analysis DuckDB tables | Generated locally by the included R scripts | | Cleaned historical website text | Download `TXT_combined.zip` from the open-access [CompuCrawl database](https://haans-mertens.github.io/data/) | A fresh run can rebuild the local DuckDB. Exact archived estimates may nevertheless differ if USITC or Compustat data have changed or if the local LLM produces different industry matches. ### Configure the website-text path Download and extract `TXT_combined.zip` from the CompuCrawl data page. Then create your local configuration: ```bash cp config_example.R config.R ``` Edit `config.R` and set: ```r WEBSITE_TEXT_DIR <- "/absolute/path/to/TXT_combined" ``` `config.R` is ignored by Git. Each user can therefore keep data wherever convenient without changing the shared analysis source. ### WRDS credentials The Jones data-preparation script reads WRDS credentials from a local `secrets.csv`: ```bash cp secrets-example.csv secrets.csv ``` Fill in `secrets.csv` locally. It is excluded by `.gitignore`. - Never commit `secrets.csv`. - Never paste credentials into an AI-agent conversation. - If WRDS authentication fails, stop. Do not retry automatically. ### Pipeline order Run commands from the repository root and validate each output before continuing: 1. Install the Python environment: ```bash poetry install ``` 2. Collect and clean public USITC data: ```bash poetry run python data-prep/Scripts/get_data.py Rscript data-prep/Scripts/select_cases.R ``` 3. Match investigations to industries with the local Ollama model: ```bash poetry run python data-prep/Scripts/industry_match_chat.py ``` 4. Download Compustat data and estimate Jones residuals: ```bash Rscript data-prep/Scripts/jones.R ``` 5. Build cohorts and analysis tables: ```bash Rscript data-prep/Scripts/phd-lecture-analyses.R ``` 6. Render only the page you intend to inspect after validating its inputs. The scripts use `data-prep/DB/jones_duckdb` as the canonical database. They create the local database and its tables during the pipeline. ## Safe starting checks These commands inspect your setup but do not install software or execute analyses: ```bash git status --short --branch git remote -v command -v git command -v quarto command -v R command -v Rscript command -v poetry command -v ollama ``` After the checks, choose one concrete target: inspect the lecture, render only the slides, rebuild the USITC stage, rebuild the Jones stage, or adapt an analysis in your fork.