{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": [
    "(working_with_local_datasets)=\n",
    "# Working with a Local Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1",
   "metadata": {},
   "source": [
    "In this tutorial, we will show how to use your own local dataset with the Dataset class. The Dataset class can help you to manage and process your eyetracking data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2",
   "metadata": {},
   "source": [
    "## Preparations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3",
   "metadata": {},
   "source": [
    "We import `pymovements` as the alias `pm` for convenience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pymovements as pm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5",
   "metadata": {},
   "source": [
    "For demonstration purposes, we will use the raw data provided by the Toy dataset, a sample dataset that comes with *pymovements*.\n",
    "\n",
    "We will download the resources of this dataset the directory to simulate a local dataset for you.\n",
    "All downloaded archive files are automatically extracted and then removed.\n",
    "The directory of the dataset will be `data/my_dataset`.\n",
    "\n",
    "After that we won't use the python class anymore and delete the object\n",
    "(the files on your system will stay in place).\n",
    "Don't worry if you're confused about these lines as they are not relevant to your use case.\n",
    "\n",
    "Just keep in mind that we now have some files with gaze data in the directory `data/my_dataset`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6",
   "metadata": {},
   "outputs": [],
   "source": [
    "toy_dataset = pm.Dataset('ToyDataset', path='data/my_dataset')\n",
    "toy_dataset.download(remove_finished=True)\n",
    "\n",
    "del toy_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7",
   "metadata": {},
   "source": [
    "## Defining your Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8",
   "metadata": {},
   "source": [
    "To load your dataset, you will need to specify a {py:class}`~pymovements.DatasetDefinition`.\n",
    "\n",
    "The following fields are required:\n",
    "\n",
    "- `name`: the (abbreviated) name of your dataset\n",
    "- `experiment`: the particular experiment setup\n",
    "- `resources`: metadata on your available dataset resources\n",
    "\n",
    "Some additional fields are optional:\n",
    "\n",
    "- `long_name`: the long-form name of your dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9",
   "metadata": {},
   "source": [
    "## Define your Experiment\n",
    "\n",
    "To use the Dataset class, we first need to create an Experiment instance. This class represents the properties of the experiment, such as the screen dimensions and sampling rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10",
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment = pm.Experiment(\n",
    "    screen_width_px=1280,\n",
    "    screen_height_px=1024,\n",
    "    screen_width_cm=38,\n",
    "    screen_height_cm=30.2,\n",
    "    distance_cm=68,\n",
    "    origin='upper left',\n",
    "    sampling_rate=1000,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11",
   "metadata": {},
   "source": [
    "## Defining your resources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12",
   "metadata": {},
   "source": [
    "Next we will define our dataset resources by setting up a {py:class}`~pymovements.ResourceDefinition`.\n",
    "\n",
    "A `ResourceDefinition` should always include the following fields:\n",
    "- `content`: the type of content (e.g., `gaze`, `precomputed_events`)\n",
    "- `filename_pattern`: the filename pattern of resource files\n",
    "\n",
    "Some additional fields are optional but might be necessary for your dataset:\n",
    "- `filename_pattern_schema_overrides`: specify datatypes of named groups in `filename_pattern`\n",
    "- `load_function`: the loading function, usually inferred automatically\n",
    "- `load_kwargs`: additional keyword arguments that are passed to the loading function\n",
    "\n",
    "In our tutorial dataset we only have one type of content: `gaze` sample data stored in csv files, hence we only need to set up a single `ResourceDefinition`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13",
   "metadata": {},
   "source": [
    "The `filename_pattern` is a pattern expression used to match dataset filenames.\n",
    "The named groups in the curly braces will be parsed as additional metadata.\n",
    "\n",
    "In our tutorial dataset all files conform to the filename pattern:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename_pattern = r'trial_{text_id:d}_{page_id:d}.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15",
   "metadata": {},
   "source": [
    "This will match filenames like `trial_1_2.csv` and parse the values of `text_id==1` and `page_id==2`.\n",
    "\n",
    "As both `text_id` and `page_id` are numeric values, we can explicitly specify the these values as `int`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename_pattern_schema_overrides = {\n",
    "    'text_id': int,\n",
    "    'page_id': int,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17",
   "metadata": {},
   "source": [
    "## Column Definitions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18",
   "metadata": {},
   "source": [
    "The `trial_columns` argument can be used to specify which columns define a single trial.\n",
    "\n",
    "This is important for correctly applying all preprocessing methods.\n",
    "\n",
    "For this tiny single user dataset a trial is just defined by `text_id` and `page_id`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19",
   "metadata": {},
   "outputs": [],
   "source": [
    "trial_columns = ['text_id', 'page_id']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20",
   "metadata": {},
   "source": [
    "The `time_column` and `pixel_columns` arguments can be used to correctly map the columns in your dataframes. If the time unit differs from the default milliseconds `ms` one must also specify the `time_unit` for correct computations.\n",
    "\n",
    "Depending on the content of your dataset, you can alternatively also provide `position_columns`, `velocity_columns` and `acceleration_columns`.\n",
    "\n",
    "Specifying these columns is needed for correctly applying preprocessing methods. For example, if you want to apply the {py:meth}`~pymovements.Dataset.pix2deg` method, you will need to specify `pixel_columns` accordingly.\n",
    "\n",
    "If your dataset has gaze positions available only in degrees of visual angle, you have to specify the `position_columns` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21",
   "metadata": {},
   "outputs": [],
   "source": [
    "time_column = 'timestamp'\n",
    "time_unit = 'ms'\n",
    "pixel_columns = ['x', 'y']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22",
   "metadata": {},
   "source": [
    "## Setting up loading function parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23",
   "metadata": {},
   "source": [
    "Now we must set up the parameters for our loading function.\n",
    "\n",
    "As the content is `gaze` and the filename extension of the `filename_pattern` is `.csv`, the loading function is automatically inferred to be `from_csv`.\n",
    "\n",
    "In case the loading function cannot be automatically inferred from your `filename_pattern` you will have to specify it explictly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24",
   "metadata": {},
   "outputs": [],
   "source": [
    "load_function = 'from_csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25",
   "metadata": {},
   "source": [
    "Have a look at the {py:func}`~pymovements.gaze.from_csv` reference to see what additional parameters you can set up.\n",
    "\n",
    "We will use our defined values for `time_column`, `time_unit` and `pixel_columns`.\n",
    "\n",
    "As our csv files are tab separated, we need to specify that separator via `load_kwargs`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26",
   "metadata": {},
   "outputs": [],
   "source": [
    "load_kwargs = {\n",
    "    'time_column': time_column,\n",
    "    'time_unit': time_unit,\n",
    "    'pixel_columns': pixel_columns,\n",
    "    'read_csv_kwargs': {'separator': '\\t'},\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27",
   "metadata": {},
   "source": [
    "We can now initialize our {py:class}`~pymovements.ResourceDefinition`. The content keyword for our gaze sample files is 'gaze'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28",
   "metadata": {},
   "outputs": [],
   "source": [
    "resource_definition = pm.ResourceDefinition(\n",
    "    content='gaze',\n",
    "    filename_pattern=filename_pattern,\n",
    "    filename_pattern_schema_overrides=filename_pattern_schema_overrides,\n",
    "    load_function=load_function,\n",
    "    load_kwargs=load_kwargs,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29",
   "metadata": {},
   "source": [
    "## Define and load the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30",
   "metadata": {},
   "source": [
    "Next we use all these definitions and create a {py:class}`~pymovements.DatasetDefinition` by passing in the root directory, Experiment instance, and other optional parameters such as the filename regular expression and custom CSV reading parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_definition = pm.DatasetDefinition(\n",
    "    name='my_dataset',\n",
    "    experiment=experiment,\n",
    "    resources=[resource_definition],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32",
   "metadata": {},
   "source": [
    "Finally, we create a {py:class}`~pymovements.Dataset` instance by using the {py:class}`~pymovements.DatasetDefinition` and specifying the directory path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pm.Dataset(\n",
    "    definition=dataset_definition,\n",
    "    path='data/my_dataset/',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34",
   "metadata": {},
   "source": [
    "If we have a root data directory which holds all your local datasets, we can further need to define the paths of the dataset.\n",
    "\n",
    "The `dataset`, `raw`, `preprocessed`, and `events` parameters define the names of the directories for the dataset, raw data, preprocessed data, and events data, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_paths = pm.DatasetPaths(\n",
    "    root='data/',\n",
    "    raw='raw',\n",
    "    preprocessed='preprocessed',\n",
    "    events='events',\n",
    ")\n",
    "\n",
    "dataset = pm.Dataset(\n",
    "    definition=dataset_definition,\n",
    "    path=dataset_paths,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36",
   "metadata": {},
   "source": [
    "Now let's load the dataset into memory. Here we select a subset including the first page of texts with ID 1 and 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37",
   "metadata": {},
   "outputs": [],
   "source": [
    "subset = {\n",
    "    'text_id': [1, 2],\n",
    "    'page_id': 1,\n",
    "}\n",
    "\n",
    "dataset.load(subset=subset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38",
   "metadata": {},
   "source": [
    "## Use the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39",
   "metadata": {},
   "source": [
    "Once we have created the Dataset instance, we can use its methods to preprocess and analyze data in our local dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.gaze[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41",
   "metadata": {},
   "source": [
    "Here we use the {py:meth}`~pymovements.Dataset.pix2deg` method to convert the pixel coordinates to degrees of visual angle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.pix2deg()\n",
    "\n",
    "dataset.gaze[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43",
   "metadata": {},
   "source": [
    "We can use the {py:meth}`~pymovements.Dataset.pos2vel` method to calculate the velocity of the gaze position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.pos2vel(method='savitzky_golay', degree=2, window_length=7)\n",
    "\n",
    "dataset.gaze[0]"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}