Reference > Paperspace > Legacy Tools > Gradient API, CLI, & SDK > Command Reference > datasets

Was this page helpful?

datasets

Validated on 14 Dec 2023 • Last edited on 15 May 2024

Note

A new Paperspace API is now available. The Gradient and Core API endpoints will become unavailable on 15 July, 2024. For more information, see the release notes.

create

Create dataset

gradient datasets create \
  --name "new_dataset_1" \
  --storageProviderId "sp65qbxcrsgiv43"

curl -X POST 'https://api.paperspace.io/datasets' \
-H 'x-api-key: d44808a2785d6a...' \
-H 'Content-Type: application/json' \
--data-raw '{
    "storageProviderId": "sp65qbxcrsgiv43",
    "name": "new_dataset_1"
}'

from gradient import DatasetsClient

api_key='d44808a2785d6a...'

datasets_client = DatasetsClient(api_key)

print(datasets_client.create(
    name='new_dataset_1',
    storage_provider_id='sp65qbxcrsgiv43'
))

Options

Name	Type	Attributes	Description
`--name`	string	required	Dataset name
`--storageProviderId`	string	required	Storage provider ID
`--description`	string	optional	Dataset description
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

Response

CLI
cURL
Python

Created dataset: ds7diuwj0av3tqy

{
  "id": "ds7diuwj0av3tqy",
  "name": "new_dataset_1",
  "description": null,
  "isPublic": false,
  "mlObjectTypeId": 2,
  "dtCreated": "2022-03-21T23:26:59.475Z",
  "dtModified": "2022-03-21T23:26:59.475Z",
  "storageProvider": {
    "name": "Gradient Managed",
    "config": {
      "bucket": "tewr3st2z",
      "endpoint": "https://osg-ny2.paperspace.io/",
      "accessKey": "6P0NXXIKSB49ZW7ZXESL",
      "retainData": false,
      "secretAccessKey": "********"
    },
    "dtCreated": "2022-03-14T15:00:11.060Z",
    "dtModified": "2022-03-14T15:00:11.060Z",
    "isTeamDefault": true,
    "isGradientManaged": true,
    "id": "sp65qbxcrsgiv43",
    "type": "s3"
  }
}

ds7diuwj0av3tqy

details

Show dataset details

Examples

CLI
cURL
Python

gradient datasets details \
  --id "ds7diuwj0av3tqy"

curl -X GET 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy' \
-H 'x-api-key: d44808a2785d6a...'

from gradient import DatasetsClient

api_key='d44808a2785d6a...'

datasets_client = DatasetsClient(api_key)

print(datasets_client.get(
    dataset_id="ds7diuwj0av3tqy"
))

Options

Name	Type	Attributes	Description
`--id`	string	required	Dataset ID
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

Response

CLI
cURL
Python

+-----------------+------------------------------------+
| Name            | new_dataset_1                      |
+-----------------+------------------------------------+
| ID              | ds7diuwj0av3tqy                    |
| Description     |                                    |
| StorageProvider | Gradient Managed (sp65qbxcrsgiv43) |
+-----------------+------------------------------------+

{
  "id": "ds7diuwj0av3tqy",
  "name": "new_dataset_1",
  "description": "Dataset description",
  "isPublic": false,
  "mlObjectTypeId": 2,
  "dtCreated": "2022-03-21T19:05:04.485Z",
  "dtModified": "2022-03-21T19:21:22.839Z",
  "storageProvider": {
    "name": "Gradient Managed",
    "config": {
      "bucket": "tewr3st2z",
      "endpoint": "https://osg-ny2.paperspace.io/",
      "accessKey": "6P0NXXIKSB49ZW7ZXESL",
      "retainData": false,
      "secretAccessKey": "********"
    },
    "dtCreated": "2022-03-14T15:00:11.060Z",
    "dtModified": "2022-03-14T15:00:11.060Z",
    "isTeamDefault": true,
    "isGradientManaged": true,
    "id": "sp65qbxcrsgiv43",
    "type": "s3"
  }
}

Dataset(
  (id = "ds7diuwj0av3tqy"),
  (name = "new_dataset_1"),
  (description = None),
  (storage_provider_id = None),
  (storage_provider = StorageProvider(
    (id = "sp65qbxcrsgiv43"),
    (type = "s3"),
    (name = "Gradient Managed"),
    (config = {
      bucket: "tewr3st2z",
      endpoint: "https://osg-ny2.paperspace.io/",
      accessKey: "6P0NXXIKSB49ZW7ZXESL",
      retainData: False,
      secretAccessKey: "********",
    })
  ))
);

list

List datasets

Examples

CLI
cURL
Python

gradient datasets list

curl -X GET 'https://api.paperspace.io/datasets' \
-H 'x-api-key: d44808a2785d6a...'

from gradient import DatasetsClient

api_key='d44808a2785d6a...'

datasets_client = DatasetsClient(api_key)

print(datasets_client.list())

Options

Name	Type	Attributes	Description
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

Response

CLI
cURL
Python

+---------------+-----------------+------------------------------------+
| Name          | ID              | Storage Provider                   |
+---------------+-----------------+------------------------------------+
| new_dataset_1 | ds7diuwj0av3tqy | Gradient Managed (sp65qbxcrsgiv43) |
+---------------+-----------------+------------------------------------+

[
  {
    id: "ds7diuwj0av3tqy",
    name: "new_dataset_1",
    description: "Dataset description",
    isPublic: false,
    mlObjectTypeId: 2,
    dtCreated: "2022-03-15T00:23:00.481Z",
    dtModified: "2022-03-15T00:49:20.202Z",
    storageProvider: {
      name: "Gradient Managed",
      config: {
        bucket: "tewr3st2z",
        endpoint: "https://osg-ny2.paperspace.io/",
        accessKey: "6P0NXXIKSB49ZW7ZXESL",
        retainData: false,
        secretAccessKey: "********",
      },
      dtCreated: "2022-03-14T15:00:11.060Z",
      dtModified: "2022-03-14T15:00:11.060Z",
      isTeamDefault: true,
      isGradientManaged: true,
      id: "sp65qbxcrsgiv43",
      type: "s3",
    },
  },
]

[
  Dataset(
    (id = "ds7diuwj0av3tqy"),
    (name = "new_dataset_1"),
    (description = "Updated dataset description"),
    (storage_provider_id = None),
    (storage_provider = StorageProvider(
      (id = "sp65qbxcrsgiv43"),
      (type = "s3"),
      (name = "Gradient Managed"),
      (config = {
        bucket: "tewr3st2z",
        endpoint: "https://osg-ny2.paperspace.io/",
        accessKey: "6P0NXXIKSB49ZW7ZXESL",
        retainData: False,
        secretAccessKey: "********",
      })
    ))
  ),
];

update

Update dataset

Examples

CLI
cURL
Python

gradient datasets update \
  --id "ds7diuwj0av3tqy" \
  --description "Updated dataset description"

curl -X POST 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy' \
-H 'x-api-key: d44808a2785d6a...' \
-H 'Content-Type: application/json' \
--data-raw '{
    "description": "Updated dataset description"
}'

from gradient import DatasetsClient

api_key='d44808a2785d6a...'

datasets_client = DatasetsClient(api_key)

print(datasets_client.update(
    dataset_id="ds7diuwj0av3tqy",
    description="Updated dataset description"
))

Options

Name	Type	Attributes	Description
`--id`	string	required	Dataset ID
`--name`	string	optional	Dataset name
`--description`	string	optional	Dataset description
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

Response

CLI
cURL
Python

Updated dataset: ds7diuwj0av3tqy

{
  "id": "ds7diuwj0av3tqy",
  "name": "new_name",
  "description": "Updated dataset description",
  "isPublic": false,
  "mlObjectTypeId": 2,
  "dtCreated": "2022-03-21T19:05:04.485Z",
  "dtModified": "2022-03-21T23:44:36.440Z",
  "storageProvider": {
    "name": "Gradient Managed",
    "config": {
      "bucket": "tewr3st2z",
      "endpoint": "https://osg-ny2.paperspace.io/",
      "accessKey": "6P0NXXIKSB49ZW7ZXESL",
      "retainData": false,
      "secretAccessKey": "********"
    },
    "dtCreated": "2022-03-14T15:00:11.060Z",
    "dtModified": "2022-03-14T15:00:11.060Z",
    "isTeamDefault": true,
    "isGradientManaged": true,
    "id": "sp65qbxcrsgiv43",
    "type": "s3"
  }
}

None

delete

Delete dataset

Examples

CLI
cURL
Python

gradient datasets delete \
  --id "dsltzdafwejmqlq"

curl -X DELETE 'https://api.paperspace.io/datasets/dsltzdafwejmqlq' \
-H 'x-api-key: d44808a2785d6a...'

from gradient import DatasetsClient

api_key='d44808a2785d6a...'

datasets_client = DatasetsClient(api_key)

print(datasets_client.delete(
    dataset_id='ds7diuwj0av3tqy',
))

Options

Name	Type	Attributes	Description
`--id`	string	required	Dataset ID
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

Response

CLI
cURL
Python

Deleted dataset: dsltzdafwejmqlq

None

files get

Get files

Examples

gradient datasets files get \
  --id "dsj1g16f1z67jai:0z7o247" \
  --target-path "~/downloads"

Options

Name	Type	Attributes	Description
`--id`	string	required	Dataset version ID
`--source-path`	string	optional	File or directory to get
`--target-path`	string	required	Target directory path
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

files list

List files

Examples

gradient datasets files list \
  --id "dsj1g16f1z67jai:xbreynt"

Options

Name	Type	Attributes	Description
`--id`	string	required	Dataset version ID
`--path`	string	optional	Sub-directory to delete
`--recursive`	string	optional	Recursive list content
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

Response

+----------+------+
| Name     | Size |
+----------+------+
| data.txt | 4    |
+----------+------+

files put

Put files

Examples

gradient datasets files put \
  --id "dsj1g16f1z67jai:xbreynt" \
  --source-path "data.txt"

Options

Name	Type	Attributes	Description
`--id`	string	required	Dataset version ID
`--source-path`	string	required	File or directory to get
`--target-path`	string	optional	Target directory path
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

files delete

Delete files

Examples

gradient datasets files delete \
  --id "dsj1g16f1z67jai:0z7o247"

Options

Name	Type	Attributes	Description
`--id`	string	required	Dataset version ID
`--path`	string	optional	Sub-directory to delete
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

versions create

Create dataset version

Examples

CLI
cURL
Python

gradient datasets versions create \
  --id "ds7diuwj0av3tqy"

curl -X POST 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy/versions' \
-H 'x-api-key: d44808a2785d6a...'

from gradient import DatasetVersionsClient

api_key = 'd44808a2785d6a...'

datasetVersions_client = DatasetVersionsClient(api_key)

print(datasetVersions_client.create(
    dataset_id="ds7diuwj0av3tqy"
))

Options

Name	Type	Attributes	Description
`--id`	string	required	Dataset version ID
`--message`	string	optional	Dataset version message
`--source-path`	string	optional	Files to put into dataset versioning (setting this commits the version)
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

Response

CLI
cURL
Python

Created dataset version: ds7diuwj0av3tqy:dk0c3k1

{
  "datasetId": "513d7d32-6a9d-4fbb-8c26-f1fd1f989ddd",
  "version": "dk0c3k1",
  "url": "s3://tewr3st2z/tewr3st2z/datasets/ds7diuwj0av3tqy/versions/dk0c3k1",
  "isCommitted": false,
  "message": null,
  "metadata": null,
  "dtCreated": "2022-03-21T23:51:56.952Z",
  "dtModified": "2022-03-21T23:51:56.952Z",
  "tags": [],
  "inputOutputs": [],
  "models": []
}

dk0c3k1

versions commit

Commit dataset version

Examples

gradient datasets versions commit \
  --id "ds7diuwj0av3tqy:dk0c3k1"

Options

Name	Type	Attributes	Description
`--id`	string	required	Dataset version ID
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

Response

Committed dataset version: ds7diuwj0av3tqy:dk0c3k1

versions details

Show dataset version details

Examples

CLI
cURL
Python

gradient datasets versions details \
  --id "ds7diuwj0av3tqy:dk0c3k1"

curl -X GET 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy/versions/dk0c3k1' \
-H 'x-api-key: d44808a2785d6a...'

from gradient import DatasetVersionsClient

api_key = 'd44808a2785d6a...'

datasetVersions_client = DatasetVersionsClient(api_key)

print(datasetVersions_client.get(
    dataset_version_id="ds7diuwj0av3tqy:dk0c3k1"
))

Options

Name	Type	Attributes	Description
`--id`	string	required	Dataset version ID
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

Response

CLI
cURL
Python

+-----------+-------------------------+
| ID        | ds7diuwj0av3tqy:dk0c3k1 |
+-----------+-------------------------+
| Message   |                         |
| Committed | true                    |
| Tags      |                         |
+-----------+-------------------------+

{
  "datasetId": "513d7d32-6a9d-4fbb-8c26-f1fd1f989ddd",
  "version": "dk0c3k1",
  "url": "s3://tewr3st2z/tewr3st2z/datasets/ds7diuwj0av3tqy/versions/dk0c3k1",
  "isCommitted": true,
  "message": null,
  "metadata": null,
  "dtCreated": "2022-03-21T23:51:56.952Z",
  "dtModified": "2022-03-22T00:03:38.917Z",
  "tags": [],
  "inputOutputs": [],
  "models": []
}

DatasetVersion(version='dk0c3k1', message=None, is_committed=True, tags=[], dataset_id=None)

versions list

List dataset versions

Examples

CLI
cURL
Python

gradient datasets versions list \
  --id "ds7diuwj0av3tqy"

curl -X GET 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy/versions' \
-H 'x-api-key: d44808a2785d6a...'

from gradient import DatasetVersionsClient

api_key = 'd44808a2785d6a...'

datasetVersions_client = DatasetVersionsClient(api_key)

print(datasetVersions_client.list(
    dataset_id="ds7diuwj0av3tqy"
))

Options

Name	Type	Attributes	Description
`--id`	string	required	Dataset ID
`--isCommitted`	boolean	optional	Show filter by committed status
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

Response

CLI
cURL
Python

+-------------------------+---------+------+
| ID                      | Message | Tags |
+-------------------------+---------+------+
| ds7diuwj0av3tqy:dk0c3k1 |         |      |
+-------------------------+---------+------+

{
  "datasetId": "513d7d32-6a9d-4fbb-8c26-f1fd1f989ddd",
  "version": "dk0c3k1",
  "url": "s3://tewr3st2z/tewr3st2z/datasets/ds7diuwj0av3tqy/versions/dk0c3k1",
  "isCommitted": true,
  "message": null,
  "metadata": null,
  "dtCreated": "2022-03-21T23:51:56.952Z",
  "dtModified": "2022-03-22T00:03:38.917Z",
  "tags": [],
  "inputOutputs": [],
  "models": []
}

[
  DatasetVersion(
    (version = "dk0c3k1"),
    (message = None),
    (is_committed = True),
    (tags = []),
    (dataset_id = None)
  ),
];

versions update

Update dataset version

Examples

CLI
cURL
Python

gradient datasets versions update \
  --id "ds7diuwj0av3tqy:dk0c3k1"

curl -X POST 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy/versions/dk0c3k1' \
-H 'x-api-key: d44808a2785d6a...'

from gradient import DatasetVersionsClient

api_key = 'd44808a2785d6a...'

datasetVersions_client = DatasetVersionsClient(api_key)

print(datasetVersions_client.update(
    dataset_version_id="ds7diuwj0av3tqy:dk0c3k1"
))

Options

Name	Type	Attributes	Description
`--id`	string	required	Dataset version ID
`--message`	string	optional	Dataset version message
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

Response

CLI
cURL
Python

Updated dataset version: ds7diuwj0av3tqy:dk0c3k1

{
  "datasetId": "513d7d32-6a9d-4fbb-8c26-f1fd1f989ddd",
  "version": "dk0c3k1",
  "url": "s3://tewr3st2z/tewr3st2z/datasets/ds7diuwj0av3tqy/versions/dk0c3k1",
  "isCommitted": true,
  "message": null,
  "metadata": null,
  "dtCreated": "2022-03-21T23:51:56.952Z",
  "dtModified": "2022-03-22T00:03:38.917Z",
  "tags": [],
  "inputOutputs": [],
  "models": []
}

None

versions delete

Delete dataset version

Examples

CLI
cURL
Python

gradient datasets versions delete \
  --id "ds7diuwj0av3tqy:dk0c3k1"

curl -X DELETE 'https://api.paperspace.io/datasets/ds7diuwj0av3tqy/versions/dk0c3k1' \
-H 'x-api-key: d44808a2785d6a...'

from gradient import DatasetVersionsClient

api_key = 'd44808a2785d6a...'

datasetVersions_client = DatasetVersionsClient(api_key)

print(datasetVersions_client.delete(
    dataset_version_id="ds7diuwj0av3tqy:8vkjy15"
))

Options

Name	Type	Attributes	Description
`--id`	string	required	Dataset version ID
`--apiKey`	string	optional	API key to use this time only
`--optionsFile`	string	optional	Path to YAML with predefined options
`--createOptionsFile`	string	optional	Generate template options file

Response

CLI
cURL
Python

Deleted dataset version: ds7diuwj0av3tqy:dk0c3k1

None

datasets

create

Examples

Options

Response

details

Examples

Options

Response

list

Examples

Options

Response

update

Examples

Options

Response

delete

Examples

Options

Response

files get

Examples

Options

files list

Examples

Options

Response

files put

Examples

Options

files delete

Examples

Options

versions create

Examples

Options

Response

versions commit

Examples

Options

Response

versions details

Examples

Options

Response

versions list

Examples

Options

Response

versions update

Examples

Options

Response

versions delete

Examples

Options

Response

We can't find any results for your search.