{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fd6678bd",
   "metadata": {},
   "source": [
    "# Reading and exploring Parquet microdata with an embedded record layout\n",
    "\n",
    "This notebook shows how to work in Python with the `.parquet` microdata files\n",
    "published by the INE (Spanish National Statistics Institute), which include:\n",
    "\n",
    "1. A columnar microdata table (Apache Parquet format).\n",
    "2. A **record layout** in JSON format embedded in the Parquet schema metadata,\n",
    "   under the `record_layout` key.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dc14a56",
   "metadata": {},
   "source": [
    "## 1. Installing dependencies\n",
    "\n",
    "Run only if not already installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43817fe3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install pyarrow pandas\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfecd4d1",
   "metadata": {},
   "source": [
    "## 2. Imports and configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab153e5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from pathlib import Path\n",
    "\n",
    "import pandas as pd\n",
    "import pyarrow.parquet as pq\n",
    "\n",
    "pd.set_option(\"display.max_columns\", 100)\n",
    "pd.set_option(\"display.width\", 140)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "642a5024",
   "metadata": {},
   "source": [
    "## 3. Downloading the INE ZIP file and locating the Parquet file\n",
    "\n",
    "The microdata ZIP file is downloaded directly from the INE website, and the\n",
    "`.parquet` file it contains is extracted, without needing to save it manually\n",
    "to disk first. Adjust `zip_url` if you want to point to a different survey or year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afacce55",
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import urllib.request\n",
    "import zipfile\n",
    "\n",
    "zip_url = \"https://www.ine.es/ftp/microdatos/eaner/datos_2025.zip\"\n",
    "download_dir = Path(\"downloaded_microdata\")\n",
    "download_dir.mkdir(exist_ok=True)\n",
    "\n",
    "# Download the ZIP file\n",
    "print(f\"Downloading {zip_url} ...\")\n",
    "with urllib.request.urlopen(zip_url) as response:\n",
    "    zip_content = response.read()\n",
    "print(f\"Download complete ({len(zip_content) / 1_048_576:.1f} MB)\")\n",
    "\n",
    "# Extract the ZIP file (in memory, without saving the .zip to disk)\n",
    "with zipfile.ZipFile(io.BytesIO(zip_content)) as zf:\n",
    "    parquet_names = [n for n in zf.namelist() if n.lower().endswith(\".parquet\")]\n",
    "\n",
    "    if not parquet_names:\n",
    "        raise FileNotFoundError(\"No .parquet file was found inside the ZIP.\")\n",
    "    if len(parquet_names) > 1:\n",
    "        print(\"Warning: there are several .parquet files in the ZIP, the first one will be used:\")\n",
    "        print(parquet_names)\n",
    "\n",
    "    parquet_name = parquet_names[0]\n",
    "    zf.extract(parquet_name, path=download_dir)\n",
    "\n",
    "parquet_path = download_dir / parquet_name\n",
    "\n",
    "assert parquet_path.exists(), f\"File not found: {parquet_path}\"\n",
    "parquet_path\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3904b24d",
   "metadata": {},
   "source": [
    "## 4. Reading the Parquet table\n",
    "\n",
    "`pyarrow.parquet` is used directly (instead of `pandas.read_parquet`) because\n",
    "this preserves access to `schema.metadata`, where the record layout is\n",
    "embedded. If loaded with pandas alone, that metadata is easily lost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f18ad36e",
   "metadata": {},
   "outputs": [],
   "source": [
    "table = pq.read_table(parquet_path)\n",
    "\n",
    "print(f\"Rows:    {table.num_rows:,}\")\n",
    "print(f\"Columns: {table.num_columns}\")\n",
    "print()\n",
    "print(\"Column names:\")\n",
    "print(table.column_names)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "805f55e6",
   "metadata": {},
   "source": [
    "## 5. Data: preview as a pandas DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8edc4c01",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = table.to_pandas()\n",
    "df.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "337663f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtypes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "326c748d",
   "metadata": {},
   "source": [
    "## 6. Metadata: extracting the `record_layout`\n",
    "\n",
    "The record layout is stored as JSON text (UTF-8) under the `record_layout`\n",
    "key in the schema metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8eceeaaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata = table.schema.metadata\n",
    "\n",
    "if metadata and b\"record_layout\" in metadata:\n",
    "    json_bytes = metadata[b\"record_layout\"]\n",
    "    json_text = json_bytes.decode(\"utf-8\")\n",
    "    record_layout = json.loads(json_text)\n",
    "    print(\"record_layout found successfully.\")\n",
    "else:\n",
    "    record_layout = None\n",
    "    print(\"'record_layout' was not found in the file metadata.\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "423f6ea4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Raw preview of the JSON (first 1000 characters)\n",
    "if record_layout is not None:\n",
    "    print(json.dumps(record_layout, indent=2, ensure_ascii=False)[:1000])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da9981ce",
   "metadata": {},
   "source": [
    "### 6.1 Interactive inspection (optional)\n",
    "\n",
    "If working in classic Jupyter (not all environments support\n",
    "`IPython.display.JSON` or widgets), this cell provides an interactive,\n",
    "collapsible view of the JSON."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4afec12a",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from IPython.display import JSON, display\n",
    "    display(JSON(record_layout))\n",
    "except Exception as e:\n",
    "    print(\"Interactive view not available in this environment:\", e)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}