# dataset module¶

the dataset module defines some tools for managing datasets.

Users may use both built-in and user-defined datasets (see the Getting Started page for examples). Right now, three built-in datasets are available:

Built-in datasets can all be loaded (or downloaded if you haven’t already) using the Dataset.load_builtin() method. For each built-in dataset, Surprise also provide predefined readers which are useful if you want to use a custom dataset that has the same format as a built-in one.

Summary:

 Dataset.load_builtin Load a built-in dataset. Dataset.load_from_file Load a dataset from a (custom) file. Dataset.load_from_folds Load a dataset where folds (for cross-validation) are predefined by some files. Dataset.folds Generator function to iterate over the folds of the Dataset. DatasetAutoFolds.split Split the dataset into folds for future cross-validation. Reader The Reader class is used to parse a file containing ratings. Trainset A trainset contains all useful data that constitutes a training set.
class surprise.dataset.Dataset(reader)

Note that you should never instantiate the Dataset class directly (same goes for its derived classes), but instead use one of the three available methods for loading datasets.

folds()

Generator function to iterate over the folds of the Dataset.

See User Guide for usage.

Yields: tuple – Trainset and testset of current fold.
classmethod load_builtin(name=u'ml-100k')

If the dataset has not already been loaded, it will be downloaded and saved. You will have to split your dataset using the split method. See an example in the User Guide.

Parameters: name (string) – The name of the built-in dataset to load. Accepted values are ‘ml-100k’, ‘ml-1m’, and ‘jester’. Default is ‘ml-100k’. A Dataset object. ValueError – If the name parameter is incorrect.
classmethod load_from_df(df, reader)

Load a dataset from a pandas dataframe.

Use this if you want to use a custom dataset that is stored in a pandas dataframe. See the User Guide for an example.

Parameters: df (Dataframe) – The dataframe containing the ratings. It must have three columns, corresponding to the user (raw) ids, the item (raw) ids, and the ratings, in this order. reader (Reader) – A reader to read the file. Only the rating_scale field needs to be specified.
classmethod load_from_file(file_path, reader)

Load a dataset from a (custom) file.

Use this if you want to use a custom dataset and all of the ratings are stored in one file. You will have to split your dataset using the split method. See an example in the User Guide.

Parameters: file_path (string) – The path to the file containing ratings. reader (Reader) – A reader to read the file.
classmethod load_from_folds(folds_files, reader)

Load a dataset where folds (for cross-validation) are predefined by some files.

The purpose of this method is to cover a common use case where a dataset is already split into predefined folds, such as the movielens-100k dataset which defines files u1.base, u1.test, u2.base, u2.test, etc... It can also be used when you don’t want to perform cross-validation but still want to specify your training and testing data (which comes down to 1-fold cross-validation anyway). See an example in the User Guide.

Parameters: folds_files (iterable of tuples) – The list of the folds. A fold is a tuple of the form (path_to_train_file, path_to_test_file). reader (Reader) – A reader to read the files.
class surprise.dataset.DatasetAutoFolds(ratings_file=None, reader=None, df=None)

A derived class from Dataset for which folds (for cross-validation) are not predefined. (Or for when there are no folds at all).

build_full_trainset()

Do not split the dataset into folds and just return a trainset as is, built from the whole dataset.

User can then query for predictions, as shown in the User Guide.

Returns: The Trainset.
split(n_folds=5, shuffle=True)

Split the dataset into folds for future cross-validation.

If you forget to call split(), the dataset will be automatically shuffled and split for 5-folds cross-validation.

You can obtain repeatable splits over your all your experiments by seeding the RNG:

import random
random.seed(my_seed)  # call this before you call split!

Parameters: n_folds (int) – The number of folds. shuffle (bool) – Whether to shuffle ratings before splitting. If False, folds will always be the same each time the experiment is run. Default is True.
class surprise.dataset.Reader(name=None, line_format=u'user item rating', sep=None, rating_scale=(1, 5), skip_lines=0)

The Reader class is used to parse a file containing ratings.

Such a file is assumed to specify only one rating per line, and each line needs to respect the following structure:

user ; item ; rating ; [timestamp]


where the order of the fields and the separator (here ‘;’) may be arbitrarily defined (see below). brackets indicate that the timestamp field is optional.

Parameters: name (string, optional) – If specified, a Reader for one of the built-in datasets is returned and any other parameter is ignored. Accepted values are ‘ml-100k’, ‘ml-1m’, and ‘jester’. Default is None. line_format (string) – The fields names, in the order at which they are encountered on a line. Default is 'user item rating'. sep (char) – the separator between fields. Example : ';'. rating_scale (tuple, optional) – The rating scale used for every rating. Default is (1, 5). skip_lines (int, optional) – Number of lines to skip at the beginning of the file. Default is 0.
class surprise.dataset.Trainset(ur, ir, n_users, n_items, n_ratings, rating_scale, offset, raw2inner_id_users, raw2inner_id_items)

A trainset contains all useful data that constitutes a training set.

It is used by the train() method of every prediction algorithm. You should not try to built such an object on your own but rather use the Dataset.folds() method or the DatasetAutoFolds.build_full_trainset() method.

ur

defaultdict of list – The users ratings. This is a dictionary containing lists of tuples of the form (item_inner_id, rating). The keys are user inner ids.

ir

defaultdict of list – The items ratings. This is a dictionary containing lists of tuples of the form (user_inner_id, rating). The keys are item inner ids.

n_users

Total number of users $$|U|$$.

n_items

Total number of items $$|I|$$.

n_ratings

Total number of ratings $$|R_{train}|$$.

rating_scale

tuple – The minimum and maximal rating of the rating scale.

global_mean

The mean of all ratings $$\mu$$.

all_items()

Generator function to iterate over all items.

Yields: Inner id of items.
all_ratings()

Generator function to iterate over all ratings.

Yields: A tuple (uid, iid, rating) where ids are inner ids (see this note).
all_users()

Generator function to iterate over all users.

Yields: Inner id of users.
build_anti_testset()

Return a list of ratings that can be used as a testset in the test() method.

The ratings are all the ratings that are not in the trainset, i.e. all the ratings $$r_{ui}$$ where the user $$u$$ is known, the item $$i$$ is known, but the rating $$r_{ui}$$ is not in the trainset. As $$r_{ui}$$ is unknown, it is assumed to be equal to the mean of all ratings global_mean.

build_testset()

Return a list of ratings that can be used as a testset in the test() method.

The ratings are all the ratings that are in the trainset, i.e. all the ratings returned by the all_ratings() generator. This is useful in cases where you want to to test your algorithm on the trainset.

global_mean

Return the mean of all ratings.

It’s only computed once.

knows_item(iid)

Indicate if the item is part of the trainset.

An item is part of the trainset if the item was rated at least once.

Parameters: iid (int) – The (inner) item id. See this note. True if item is part of the trainset, else False.
knows_user(uid)

Indicate if the user is part of the trainset.

A user is part of the trainset if the user has at least one rating.

Parameters: uid (int) – The (inner) user id. See this note. True if user is part of the trainset, else False.
to_inner_iid(riid)

Convert an item raw id to an inner id.

See this note.

Parameters: riid (str) – The item raw id. The item inner id. int ValueError – When item is not part of the trainset.
to_inner_uid(ruid)

Convert a user raw id to an inner id.

See this note.

Parameters: ruid (str) – The user raw id. The user inner id. int ValueError – When user is not part of the trainset.
to_raw_iid(iiid)

Convert an item inner id to a raw id.

See this note.

Parameters: iiid (int) – The item inner id. The item raw id. str ValueError – When iiid is not an inner id.
to_raw_uid(iuid)

Convert a user inner id to a raw id.

See this note.

Parameters: iuid (int) – The user inner id. The user raw id. str ValueError – When iuid is not an inner id.