{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "provenance": [] }, "kernelspec": { "name": "ir", "display_name": "R" }, "language_info": { "name": "R" } }, "cells": [ { "cell_type": "markdown", "source": [ "# **Uso de ficheros Parquet de microdatos con diseño de registro embebido**" ], "metadata": { "id": "NpdEPHQnnk7k" } }, { "cell_type": "markdown", "source": [ "## 1. DESCRIPCIÓN GENERAL\n", "\n", "Los ficheros `.parquet` incluidos en los microdatos publicados por el INE contienen:\n", "\n", "1. Una tabla de microdatos columnar en formato Apache Parquet.\n", "2. Un diseño de registro en formato JSON embebido dentro de los metadatos del esquema Parquet.\n", "\n", "El JSON se almacena bajo la clave `record_layout` de forma que el diseño de registro permita describir variables, formatos, etiquetas, longitudes, tipos u otra documentación estructural del fichero." ], "metadata": { "id": "NqJ54nfEFToK" } }, { "cell_type": "markdown", "source": [ "\n", "\n", "---\n", "\n", "\n", "## 2. SOFTWARE COMPATIBLE\n", "\n", "Los ficheros pueden abrirse desde múltiples herramientas:\n", "\n", "* R / RStudio\n", "* Python\n", "* Apache Spark\n", "* DuckDB\n", "* VS Code\n", "* DBeaver\n", "* Tableau\n", "* Power BI\n", "* Excel (mediante conectores)\n", "* Herramientas Arrow/Parquet compatibles\n" ], "metadata": { "id": "cCw9SYkxGl5h" } }, { "cell_type": "markdown", "source": [ "\n", "\n", "---\n", "\n", "\n", "## 3. USO EN R / RSTUDIO\n", "\n", "Instalación de paquetes necesarios:\n", "\n", "\n" ], "metadata": { "id": "rdcOZxdMG86a" } }, { "cell_type": "code", "source": [ "install.packages(\"arrow\")\n", "install.packages(\"jsonlite\")\n", "install.packages(\"listviewer\")\n" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 0 }, "collapsed": true, "id": "acvef2GWHJpz", "outputId": "132b360f-71d1-4a35-8427-d5955e742e64" }, "execution_count": 13, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "Installing package into ‘/usr/local/lib/R/site-library’\n", "(as ‘lib’ is unspecified)\n", "\n", "Warning message in install.packages(\"arrow\"):\n", "“installation of package ‘arrow’ had non-zero exit status”\n", "Installing package into ‘/usr/local/lib/R/site-library’\n", "(as ‘lib’ is unspecified)\n", "\n" ] } ] }, { "cell_type": "markdown", "source": [ "Lectura de las librerías instaladas:" ], "metadata": { "id": "bkr054iIKY-T" } }, { "cell_type": "code", "source": [ "library(arrow)\n", "library(jsonlite)\n", "library(listviewer)" ], "metadata": { "id": "9NsxKIMNKglS" }, "execution_count": 14, "outputs": [] }, { "cell_type": "markdown", "source": [ "### 3.1. Acceder al fichero Parquet\n", "\n", "Introducir la ruta al fichero buscado. Se tomará como ejemplo el fichero de microdatos en el formato `.parquet` de la encuesta ANER 2025." ], "metadata": { "id": "7_AAds97IAM6" } }, { "cell_type": "code", "metadata": { "id": "deeb96c9" }, "source": [ "ruta_zip <- \"https://www.ine.es/ftp/microdatos/eaner/datos_2025.zip\"\n", "nombre_local_zip <- \"datos_2025.zip\"\n", "\n", "# Descargar el archivo ZIP\n", "download.file(ruta_zip, nombre_local_zip, mode = \"wb\")\n" ], "execution_count": 10, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "8af4ed14" }, "source": [ "Ahora, descomprimiremos el archivo ZIP:" ] }, { "cell_type": "code", "metadata": { "id": "d820cbca" }, "source": [ "# Descomprimir el archivo ZIP\n", "unzip(nombre_local_zip, exdir = \".\")\n", "\n", "# Ruta al archivo Parquet descomprimido\n", "ruta_parquet_descomprimido <- \"md_ANER_2025.parquet\" # Ajustar según la encuesta\n" ], "execution_count": 11, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "2aea6b39" }, "source": [ "### 3.2. Lectura de datos\n", "\n", "Ya se puede leer el archivo Parquet descomprimido como tabla Arrow (MUY IMPORTANTE) :" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 0 }, "collapsed": true, "id": "83476c32", "outputId": "6b628865-b026-4605-9806-6d8e9d217ac2" }, "source": [ "# Leer el archivo Parquet\n", "tabla_arrow <- arrow::read_parquet(ruta_parquet_descomprimido,\n", " as_data_frame = FALSE\n", " )\n", "\n", "# Verificar el contenido de la tabla Arrow\n", "print(tabla_arrow)\n" ], "execution_count": 12, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Table\n", "299732 rows x 7 columns\n", "$ANNO \n", "$CPRO \n", "$SEXO \n", "$EDAD \n", "$NACIONALIDAD \n", "$PAIS_NACIMIENTO \n", "$MODO \n", "\n", "See $metadata for additional Schema metadata\n" ] } ] }, { "cell_type": "markdown", "source": [ "### 3.3. Acceso a microdatos y metadatos\n", "\n", "Se crean dos objetos distintos, el primero de ellos con la información de los microdatos:" ], "metadata": { "id": "tpO-P3MTN1Fx" } }, { "cell_type": "code", "source": [ "tabla_microdatos <- as.data.frame(tabla_arrow)" ], "metadata": { "id": "IKJv5B1DOe2Y" }, "execution_count": 15, "outputs": [] }, { "cell_type": "markdown", "source": [ "Y el otro de ellos con la información de los metadatos. Lo que permite una visualización interactiva del JSON con el diseño de registro:" ], "metadata": { "id": "SJTMzlxiOicJ" } }, { "cell_type": "code", "source": [ "library(listviewer)\n", "json_guardado <- tabla_arrow$schema$metadata$record_layout\n", "jsonedit(json_guardado)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 517 }, "collapsed": true, "id": "IgYszI9YO2hg", "outputId": "bacbb216-4c14-40cc-d2d3-39038cc8a7bb" }, "execution_count": 18, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "\n", "\n", "\t\n", "\t\t\n", "\t\t\n", "\n", "\n", "\n", "\t\n", "\t\n", "\t\t
\n", "\n", "\t\n", "\n" ], "text/plain": [ "HTML widgets cannot be represented in plain text (need html)" ] }, "metadata": { "text/html": { "isolated": true } } } ] }, { "cell_type": "markdown", "source": [ "\n", "\n", "---\n", "\n", "\n", "## 4. RECOMENDACIONES Y ACLARACIONES\n", "\n", "A la hora de leer el archivo Parquet, es muy importante incluir la opción `as_data_frame = FALSE` ya que, si el Parquet se carga directamente como un `dataframe`, los metadatos internos del esquema podrían perderse. Se recomienda utilizar Arrow para preservar metadatos.\n", "\n", "El formato Parquet está optimizado para:\n", "\n", " * grandes volúmenes\n", " * lectura columnar\n", " * compresión eficiente\n", " * interoperabilidad analítica\n", "\n" ], "metadata": { "id": "kIl3T3PTQ5Fb" } }, { "cell_type": "markdown", "source": [ "\n", "\n", "---\n", "\n", "\n", "## 5. TECNOLOGÍAS UTILIZADAS\n", "\n", "* Apache Parquet\n", "* Apache Arrow\n", "* R Arrow\n", "* JSON\n" ], "metadata": { "id": "-qM518kHljQT" } } ] }