{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "provenance": [] }, "kernelspec": { "name": "ir", "display_name": "R" }, "language_info": { "name": "R" } }, "cells": [ { "cell_type": "markdown", "source": [ "# **Using Microdata Parquet Files with Embedded Record Layout**" ], "metadata": { "id": "NpdEPHQnnk7k" } }, { "cell_type": "markdown", "source": [ "## 1. OVERVIEW\n", "\n", "The `.parquet` files included in the microdata published by INE contain:\n", "\n", "1. A columnar microdata table in Apache Parquet format.\n", "2. A record layout in JSON format embedded within the Parquet schema metadata.\n", "\n", "The JSON is stored under the `record_layout` key, allowing the record layout to describe variables, formats, labels, lengths, types, or other structural documentation of the file." ], "metadata": { "id": "NqJ54nfEFToK" } }, { "cell_type": "markdown", "source": [ "\n", "---\n", "\n", "\n", "## 2. COMPATIBLE SOFTWARE\n", "\n", "The files can be opened with multiple tools:\n", "\n", "* R / RStudio\n", "* Python\n", "* Apache Spark\n", "* DuckDB\n", "* VS Code\n", "* DBeaver\n", "* Tableau\n", "* Power BI\n", "* Excel (via connectors)\n", "* Compatible Arrow/Parquet tools" ], "metadata": { "id": "cCw9SYkxGl5h" } }, { "cell_type": "markdown", "source": [ "\n", "---\n", "\n", "\n", "## 3. USAGE IN R / RSTUDIO\n", "\n", "Installation of necessary packages:" ], "metadata": { "id": "rdcOZxdMG86a" } }, { "cell_type": "code", "source": [ "install.packages(\"arrow\")\n", "install.packages(\"jsonlite\")\n", "install.packages(\"listviewer\")\n" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "collapsed": true, "id": "acvef2GWHJpz", "outputId": "132b360f-71d1-4a35-8427-d5955e742e64" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "Installing package into ‘/usr/local/lib/R/site-library’\n", "(as ‘lib’ is unspecified)\n", "\n", "Warning message in install.packages(\"arrow\"):\n", "“installation of package ‘arrow’ had non-zero exit status”\n", "Installing package into ‘/usr/local/lib/R/site-library’\n", "(as ‘lib’ is unspecified)\n", "\n" ] } ] }, { "cell_type": "markdown", "source": [ "Reading the installed libraries:" ], "metadata": { "id": "bkr054iIKY-T" } }, { "cell_type": "code", "source": [ "library(arrow)\n", "library(jsonlite)\n", "library(listviewer)" ], "metadata": { "id": "9NsxKIMNKglS" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "### 3.1. Accessing the Parquet file\n", "\n", "Enter the path to the desired file. The microdata file in `.parquet` format from the ANER 2025 survey will be used as an example." ], "metadata": { "id": "7_AAds97IAM6" } }, { "cell_type": "code", "metadata": { "id": "deeb96c9" }, "source": [ "ruta_zip <- \"https://www.ine.es/ftp/microdatos/eaner/datos_2025.zip\"\n", "nombre_local_zip <- \"datos_2025.zip\"\n", "\n", "# Download the ZIP file\n", "download.file(ruta_zip, nombre_local_zip, mode = \"wb\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "8af4ed14" }, "source": [ "Now, we will decompress the ZIP file:" ] }, { "cell_type": "code", "metadata": { "id": "d820cbca" }, "source": [ "# Unzip the ZIP file\n", "unzip(nombre_local_zip, exdir = \".\")\n", "\n", "# Path to the unzipped Parquet file\n", "ruta_parquet_descomprimido <- \"md_ANER_2025.parquet\" # Adjust according to the survey" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "2aea6b39" }, "source": [ "### 3.2. Data Reading\n", "\n", "The unzipped Parquet file can now be read as an Arrow table (VERY IMPORTANT):" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "collapsed": true, "id": "83476c32", "outputId": "6b628865-b026-4605-9806-6d8e9d217ac2" }, "source": [ "# Read the Parquet file\n", "tabla_arrow <- arrow::read_parquet(ruta_parquet_descomprimido,\n", " as_data_frame = FALSE\n", " )\n", "\n", "# Verify the content of the Arrow table\n", "print(tabla_arrow)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Table\n", "299732 rows x 7 columns\n", "$ANNO \n", "$CPRO \n", "$SEXO \n", "$EDAD \n", "$NACIONALIDAD \n", "$PAIS_NACIMIENTO \n", "$MODO \n", "\n", "See $metadata for additional Schema metadata\n" ] } ] }, { "cell_type": "markdown", "source": [ "### 3.3. Accessing Microdata and Metadata\n", "\n", "Two distinct objects are created, the first one with the microdata information:" ], "metadata": { "id": "tpO-P3MTN1Fx" } }, { "cell_type": "code", "source": [ "tabla_microdatos <- as.data.frame(tabla_arrow)" ], "metadata": { "id": "IKJv5B1DOe2Y" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "And the other one with the metadata information. This allows an interactive visualization of the JSON with the record layout:" ], "metadata": { "id": "SJTMzlxiOicJ" } }, { "cell_type": "code", "source": [ "library(listviewer)\n", "json_guardado <- tabla_arrow$schema$metadata$record_layout\n", "jsonedit(json_guardado)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 517 }, "collapsed": true, "id": "IgYszI9YO2hg", "outputId": "bacbb216-4c14-40cc-d2d3-39038cc8a7bb" }, "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "\n", "\n", "\t\n", "\t\t\n", "\t\t\n", "\n", "\n", "\n", "\t\n", "\t\n", "\t\t
\n", "\n", "\t\n", "\n" ], "text/plain": [ "HTML widgets cannot be represented in plain text (need html)" ] }, "metadata": { "text/html": { "isolated": true } } } ] }, { "cell_type": "markdown", "source": [ "\n", "---\n", "\n", "\n", "## 4. RECOMMENDATIONS AND CLARIFICATIONS\n", "\n", "When reading the Parquet file, it is very important to include the `as_data_frame = FALSE` option because if the Parquet is loaded directly as a `dataframe`, the internal schema metadata could be lost. It is recommended to use Arrow to preserve metadata.\n", "\n", "The Parquet format is optimized for:\n", "\n", " * large volumes\n", " * columnar reading\n", " * efficient compression\n", " * analytical interoperability" ], "metadata": { "id": "kIl3T3PTQ5Fb" } }, { "cell_type": "markdown", "source": [ "\n", "---\n", "\n", "\n", "## 5. TECHNOLOGIES USED\n", "\n", "* Apache Parquet\n", "* Apache Arrow\n", "* R Arrow\n", "* JSON" ], "metadata": { "id": "-qM518kHljQT" } } ] }