{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": [
    "(contributing_datasets)=\n",
    "# Contributing Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1",
   "metadata": {},
   "source": [
    "We encourage all people to contribute their eye-tracking datasets to the pymovements library.\n",
    "This increases the visibility and impact of the associated research as included datasets can be\n",
    "easily discovered and downloaded using the pymovements software.\n",
    "\n",
    "**This contributing guide will provide you with all resources you need to contribute a dataset\n",
    "to the pymovements library. It is offered on three levels. Pick the one that corresponds best to your prior knowledge:**\n",
    "\n",
    "- **{ref}`adding_dataset_basic`:** You have never used pymovements before and don't have any programming experience.\n",
    "  → You will learn how to create an issue that will allow pymovement maintainers to add your dataset\n",
    "\n",
    "- **{ref}`adding_dataset_intermediate`:** You have some experience with programming, but are not familiar with Git.\n",
    "  → You will learn how to create a pull request with a draft of your dataset definition file\n",
    "\n",
    "- **{ref}`adding_dataset_advanced`:** You are proficient in Python and familiar with Git.\n",
    "  → You will learn how to add and test your dataset definition on your local machine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2",
   "metadata": {},
   "source": [
    "## Prerequisites: hosting your dataset\n",
    "\n",
    "**pymovements does not _host_ datasets**, it only provides an interface for downloading and reading them. Therefore, you will need to upload your dataset somewhere, such that pymovements will be able to download it.\n",
    "\n",
    "- Your data must be openly available and downloadable from a simple link without requiring additional steps like logging in. We recommend [OSF](https://osf.io/) for hosting your files, but other platforms like Zenodo or GitHub will also work.\n",
    "- Your data must be stored in one of the supported formats: CSV, ASC (EyeLink), IPC/Feather.\n",
    "- Your dataset may consist of multiple files, including ZIP files containing nested folders.\n",
    "- Trial information (e.g., trial or participant IDs) may be stored as additional columns in the data files, in the filenames, or as messages in ASC files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3",
   "metadata": {},
   "source": [
    "(adding_dataset_basic)=\n",
    "## Basic\n",
    "\n",
    "To add your dataset, we will need some information on where and in what format you stored your data, as well as some metadata about your data collection. Specifically, we need:\n",
    "\n",
    "- Links to your data files (containing sample-based data, event-based data, and/or aggregated measures)\n",
    "- Information on where/how participant IDs, trial IDs, and related data are stored (within the data files, or in the filename)\n",
    "- Information on the screen you used to present the stimuli:\n",
    "  - Screen size in centimeters\n",
    "  - Screen resolution in pixels\n",
    "  - Eye-to-screen distance\n",
    "- Information on the eye-tracker you used:\n",
    "  - Model and manufacturer\n",
    "  - Sampling rate\n",
    "  - Where the origin (0, 0) of the gaze coordinates recorded by the eye-tracker is (e.g., top left of screen, center of screen)\n",
    "- Any paper(s) you would like to be referenced by users of your dataset\n",
    "\n",
    "Once you have all the information, you can [create an issue in the pymovements repository on GitHub](https://github.com/pymovements/pymovements/issues/new?template=DATASET.md). You need a GitHub account to do this. If you prefer not creating a GitHub account, please send the information above to pymovements@python.org instead.\n",
    "\n",
    "After receiving your information, we will start working to include your dataset. It is likely that we will need some additional information from you, so please keep an eye on the GitHub issue. Once the inclusion is completed, your dataset will be included in the next release of pymovements. This process may take several weeks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4",
   "metadata": {},
   "source": [
    "(adding_dataset_intermediate)=\n",
    "## Intermediate\n",
    "\n",
    "To add a new dataset to the library, you will need to create a {py:class}`~pymovements.DatasetDefinition`. This is a text file in the YAML format that contains information about where your dataset is hosted, what format it is stored in, how it was collected, and other metadata. You can find some examples of YAML files for existing datasets [here](https://github.com/pymovements/pymovements/tree/main/src/pymovements/datasets).\n",
    "\n",
    "You will need to draft a new YAML file and create a pull request on GitHub. This requires a GitHub account. You can use this link to create a new file directly in your browser: [**NEW YAML FILE**](https://github.com/pymovements/pymovements/new/main/src/pymovements/datasets?filename=my_dataset.yaml&value=name%3A%20MyDataset%0A%0Along_name%3A%20%22Long%20name%20of%20my%20dataset%22%0A%0Aresources%3A%0A%20%20-%20content%3A%20gaze%0A%20%20%20%20url%3A%20%22https%3A%2F%2Furl.to%2Fdata%2Ffile%2Fgaze.csv%22%0A%20%20%20%20filename%3A%20%22gaze.csv%22%0A%20%20%20%20md5%3A%20%22%3Chash%20of%20file%20content%3E%22%0A%0Aexperiment%3A%0A%20%20-%20screen_width_px%3A%201920%0A%20%20-%20screen_height_px%3A%201080%0A%20%20-%20screen_width_cm%3A%2050%0A%20%20-%20screen_height_cm%3A%2028%0A%20%20-%20distance_cm%3A%2060%0A%20%20-%20origin%3A%20%22upper%20left%22%0A%20%20-%20sampling_rate%3A%201000)\n",
    "\n",
    "The most important fields are `name`, `long_name`, `resources`, which contains the links to the data files, and `experiment`, which contains metadata about the physical setup and the eye tracker:\n",
    "\n",
    "```yaml\n",
    "name: MyDataset\n",
    "\n",
    "long_name: \"Long name of my dataset\"\n",
    "\n",
    "resources:\n",
    "  - content: gaze\n",
    "    url: \"https://url.to/data/file/gaze.csv\"\n",
    "    filename: \"gaze.csv\"\n",
    "    md5: \"<MD5 hash of file content>\"\n",
    "\n",
    "experiment:\n",
    "  - eyetracker:\n",
    "    - sampling_rate: 1000\n",
    "  - screen:\n",
    "    - width_px: 1920\n",
    "    - height_px: 1080\n",
    "    - width_cm: 50\n",
    "    - height_cm: 28\n",
    "    - distance_cm: 60\n",
    "    - origin: \"upper left\"\n",
    "```\n",
    "\n",
    "The field {py:attr}`~pymovements.DatasetDefinition.resources` contains a list of {py:class}`~pymovements.ResourceDefinition` instances, which contain the necessary data to download and load a specific resource (group) of a dataset. It also includes the type of content that is contained in files of that particular resource group. In our example we only have a single type of resource: ``gaze`` (samples). Other supported content types are ``precomputed_events`` and ``precomputed_reading_measures``.\n",
    "\n",
    "Detailed documentation on the different fields can be found in the API references for {py:class}`~pymovements.DatasetDefinition` and {py:class}`~pymovements.ResourceDefinition`.\n",
    "\n",
    "To get the MD5 hash of a data file, you can either use the command line:\n",
    "\n",
    "```bash\n",
    "md5sum path/to/gaze.csv\n",
    "```\n",
    "\n",
    "or Python code:\n",
    "\n",
    "```py\n",
    "from pymovements.datasets._utils._downloads import _calculate_md5\n",
    "_calculate_md5(\"path/to/gaze.csv\")\n",
    "```\n",
    "\n",
    "After adding your information in the file, click \"Commit changes\" to create a pull request. Feel free to create a pull request even if the file is still missing some information. You (or pymovements maintainers) will still be able to edit it later. If you are unsure about something, just add a comment on the pull request, and we will help you.\n",
    "\n",
    "Once the pull request is created, we will start working to include your dataset. It is likely that we will need some additional information from you, so please keep an eye on the pull request. Once the pull request is completed and merged, your dataset will be included in the next release of pymovements. This process may take several weeks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5",
   "metadata": {},
   "source": [
    "(adding_dataset_advanced)=\n",
    "## Advanced\n",
    "\n",
    "Follow the [contributing guide](https://github.com/pymovements/pymovements/blob/main/CONTRIBUTING.md) to set up your development environment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6",
   "metadata": {},
   "source": [
    "### Setting up your dataset locally\n",
    "\n",
    "We recommend setting up and testing your dataset locally first. Please refer to the {ref}`working_with_local_datasets` tutorial.\n",
    "\n",
    "The {py:class}`~pymovements.DatasetDefinition` that we get from that tutorial looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pymovements as pm\n",
    "\n",
    "experiment = pm.Experiment(\n",
    "    sampling_rate=1000,\n",
    "    screen=pm.Screen(\n",
    "        width_px=1280,\n",
    "        height_px=1024,\n",
    "        width_cm=38,\n",
    "        height_cm=30.2,\n",
    "        distance_cm=68,\n",
    "        origin='upper left',\n",
    "    ),\n",
    ")\n",
    "\n",
    "dataset_definition = pm.DatasetDefinition(\n",
    "    name='MyDataset',\n",
    "    experiment=experiment,\n",
    "    resources=[\n",
    "        pm.ResourceDefinition(\n",
    "            content='gaze',\n",
    "            filename_pattern=r'trial_{text_id:d}_{page_id:d}.csv',\n",
    "            filename_pattern_schema_overrides={\n",
    "                'text_id': int, 'page_id': int,\n",
    "            },\n",
    "            load_kwargs={\n",
    "                'read_csv_kwargs': {'separator': '\\t'},\n",
    "                'time_column': 'timestamp',\n",
    "                'time_unit': 'ms',\n",
    "                'pixel_columns': ['x', 'y'],\n",
    "            },\n",
    "        ),\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8",
   "metadata": {},
   "source": [
    "### Adding resource definitions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9",
   "metadata": {},
   "source": [
    "The dataset definition above enables you to load your *local* data files into pymovements data structures. However, in order to download online dataset resources through pymovements we need to add the source url to {py:class}`~pymovements.ResourceDefinition`s.\n",
    "\n",
    "Let's add the URL and checksum to the {py:class}`~pymovements.DatasetDefinition` of our toy dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://github.com/pymovements/pymovements-toy-dataset/archive/refs/heads/main.zip'\n",
    "\n",
    "dataset_definition = pm.DatasetDefinition(\n",
    "    name='MyDataset',\n",
    "    resources=[\n",
    "        pm.ResourceDefinition(\n",
    "            content='gaze',\n",
    "            url=url,\n",
    "            filename='pymovements-toy-dataset.zip',\n",
    "            md5='256901852c1c07581d375eef705855d6',\n",
    "            filename_pattern=r'trial_{text_id:d}_{page_id:d}.csv',\n",
    "            filename_pattern_schema_overrides={\n",
    "                'text_id': int, 'page_id': int,\n",
    "            },\n",
    "            load_kwargs={\n",
    "                'read_csv_kwargs': {'separator': '\\t'},\n",
    "                'time_column': 'timestamp',\n",
    "                'time_unit': 'ms',\n",
    "                'pixel_columns': ['x', 'y'],\n",
    "            },\n",
    "        ),\n",
    "    ],\n",
    "    experiment=experiment,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11",
   "metadata": {},
   "source": [
    "Note that some of the information previously defined at the definition level (`has_files`, `filename_pattern`, `filename_patterns_schema_overrides`) are now defined at the level of individual resources.\n",
    "\n",
    "To get the MD5 hash of a file, you can either use the command line:\n",
    "\n",
    "```bash\n",
    "md5sum path/to/pymovements-toy-dataset.zip\n",
    "```\n",
    "\n",
    "or Python code:\n",
    "\n",
    "```py\n",
    "from pymovements.datasets._utils._downloads import _calculate_md5\n",
    "_calculate_md5(\"path/to/pymovements-toy-dataset.zip\")\n",
    "```\n",
    "\n",
    "Let's test if the data files can be downloaded and loaded into memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pm.Dataset(\n",
    "    definition=dataset_definition,\n",
    "    path='data/my_dataset',\n",
    ")\n",
    "dataset.download()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13",
   "metadata": {},
   "source": [
    "And load the dataset into memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15",
   "metadata": {},
   "source": [
    "### Writing the YAML file\n",
    "\n",
    "All public datasets in the library are defined in YAML files (see [here](https://github.com/pymovements/pymovements/tree/main/src/pymovements/datasets) for examples). These YAML files contain exactly the same fields as the {py:class}`~pymovements.DatasetDefinition` objects, and the two can be easily converted into each other.\n",
    "\n",
    "Let's convert the {py:class}`~pymovements.DatasetDefinition` of our toy dataset to a YAML file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_definition.to_yaml('my_dataset.yaml')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17",
   "metadata": {},
   "source": [
    "Let's check the content of the written file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('my_dataset.yaml', encoding='utf-8') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19",
   "metadata": {},
   "source": [
    "This YAML file can now be added to `src/pymovements/datasets/`. Commit the file in your fork of the pymovements repository and create a pull request. We will then review the pull request and request additional information or changes if necessary, so please keep an eye on the pull request. Once the pull request is completed and merged, your dataset will be included in the next release of pymovements. This process may take several weeks.\n",
    "\n",
    "If you run into problems, feel free to create a draft pull request and explain the issue so that we can provide support."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20",
   "metadata": {},
   "source": [
    "### Running integration tests\n",
    "\n",
    "To check whether the integration in the dataset library is working properly, you can run the integration test for your dataset:\n",
    "\n",
    "```bash\n",
    "tox -e integration -- \\\n",
    "  'tests/integration/public_dataset_processing_test.py::test_public_dataset_processing[my_dataset]'\n",
    "```"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}