ACRONYMS National Statistics Institute

Anonymized microdata

This page lists the statistical operations for which anonymised microdata files are available for download from the website. 

Microdata files contain individual records from a statistical operation, conveniently anonymised, in order to preserve the confidentiality of the information. 

The raw microdata files are fixed-width plain-text files containing, for each individual survey record, the value of each variable. To interpret these files correctly, it is necessary to map the variables and their values according to the specifications set out in the Excel files that contain the record layout. As the data are unaggregated, analysing them requires appropriate data-processing and analysis tools.

To make working with raw microdata easier, processed files are also provided in the following formats: CSV, R, SAS, Stata and SPSS. The Parquet format and a JSON record layout are also being introduced.

The JSON record layout lets users read the raw microdata files easily, without having to manually map the variables to their positions within the fixed-width file.

The Parquet file has the JSON record layout built in, making it self-contained and removing the need for additional files for its use.

  • Description of parquet format

    • Parquet files are a data storage format specifically designed for massive analysis systems. Unlike traditional row-based formats such as CSV and JSON, Parquet is a columnar format, meaning that it stores the data divided into columns instead of by entire records. This feature can significantly optimise access to the information, given that analysis systems usually only require a subset of variables (columns), thus avoiding the unnecessary reading of complete data. Moreover, Parquet was developed within the Hadoop ecosystem, and it’s widely integrated into modern tools such as Apache Spark, Hive and cloud processing systems.

      One of Parquet’s main technical advantages is its capacity for efficient compression and encoding. By storing data by columns, it’s easier to apply advanced compression techniques that benefit from the similarity of the values in each column. For example, columns with repetitive values or low cardinality can be compressed much more than in traditional formats. This is reflected in a significant reduction in the size of the files, leading to lower storage and transmission costs, as well as a better performance when it comes to reading and processing the data.

      Within the context of the publication of anonymised microdata, the Parquet format brings highly significant benefits. Firstly, it provides more efficient accessibility for researchers and analysts, as they can view just the necessary variables without having to load the entire dataset. The above can prove particularly useful during work with large and complex databases, such as those from surveys, administrative records and censuses.

      Furthermore, Parquet’s metadata support facilitates the structured documentation of the variables, thus improving the data’s interpretability and reuse. In the case of the files published by the INE, the metadata are incorporated into the parquet file in JSON format, complying with the registration design defined in the corresponding JSON Schema.

      Another key advantage is the interoperability with modern analytical environments. Publishing microdata in Parquet format allows for their direct use on distributed analysis platforms and in languages such as Python and R, using libraries like Pandas, PyArrow and dplyr. This removes the need for prior conversions and streamlines the users’ workflow.

      In short, the use of the Parquet format for publishing anonymised microdata combines technical efficiency, resource savings, ease of use and compatibility with modern technologies. The publication of these files seeks to promote the opening of data in a secure and efficient manner.

The National Statistics Institute (INE) accepts no responsibility for any results users obtain from these files through their own calculations. Furthermore, anyone using microdata files undertakes to cite INE as the source of the primary data in any publication resulting from their use (source: INE, www.ine.es), and to acknowledge that the accuracy or reliability of any information derived from their own processing is their sole responsibility.

Statistical operations that have available microdata files with standard anonymization: