Delete Data from an Index Layer

You can delete data and metadata from an index layer using RSQL queries. Indexed records (the metadata) which meet your deleteQuery criteria will be deleted from your index layer. Deleting the indexed records will also automatically delete the respective data stored in blob after one day.

Deletion of index layer metadata and data is an asynchronous and relatively slow process. For example, a delete request with 1 million records satisfying the deleteQuery could take 1 hour to complete deletion of metadata; The actual data in blobstore will continue to be available for 24 hours after complete deletion of metadata, then they will be completely purged.

If you want to remove all data in an index layer which contain millions of record, it would be faster to Delete A Layer and recreate it.

Note

Delete is an asynchronous operation. After you submit the delete request, the delete action could run for hours based on your delete scale. You can query the delete status while the delete action is ongoing.

To delete index layer data and metadata:

  1. Obtain an authorization token.
  2. Use the api-lookup API to get API base URLs.
  3. Use the index API to delete the indexed records along with the corresponding data that matches your delete query.

Obtain an Authorization Token

Obtain an authorization token for your HTTP requests. For instructions, see the Authentication and Authorization Developer's Guide.

Get API Base URLs

Use the API Lookup service to get the API endpoints for the index API of the catalog containing the index layer with data handles you want to delete.

For instructions, see the API Lookup Developer's Guide.

Delete Data

In order to delete data, you must perform Delete action by specifying "deleteQuery" request parameter in the index API.

DELETE /<Base path for the index API from the API Lookup Service>/layers/<Layer ID>?deleteQuery=<RSQL> HTTP/1.1
Host: <Hostname for the index API from the API Lookup Service>
Authorization: Bearer <Authorization Token>
Accept: application/json
Cache-Control: no-cache

The rule for \ in "deleteQuery" request parameter is the same as the "query" request parameter in Get Data from an Index Layer -> Section "Get the Data Handle".

Once you perform the Delete action above, you will receive a deleteId in the response. Example response:

{
    "deleteId": "f054dd41-51f2-4633-9a42-cbcf59cc24e1"
}

Query Delete Status

You can use the deleteId obtained above to query the status of the delete action you just submitted:

GET /<Base path for the index API from the API Lookup Service>/layers/<Layer ID>/deleteRequest/<Delete ID> HTTP/1.1
Host: <Hostname for the index API from the API Lookup Service>
Authorization: Bearer <Authorization Token>
Accept: application/json
Cache-Control: no-cache

You will receive one of the 4 possible statuses:

  1. Scheduled
  2. Processing
  3. Succeeded
  4. Failed

Examples of responses:

{
    "deleteId": "f054dd41-51f2-4633-9a42-cbcf59cc24e1",
    "state": "Scheduled",
    "message": "The delete request was successfuly scheduled.",
    "count": 0
}

When the state is "Scheduled", the count is always 0, meaning no index record deleted yet.

{
    "deleteId": "f054dd41-51f2-4633-9a42-cbcf59cc24e1",
    "state": "Processing",
    "message": "The delete request is being executed.",
    "count": 7289
}

When the state is "Processing", the count is the number of index records that have been deleted so far.

{
    "deleteId": "f054dd41-51f2-4633-9a42-cbcf59cc24e1",
    "state": "Succeeded",
    "message": "The delete request finished successfuly. The number of deleted records is provided in the `count` field.",
    "count": 13445
}

When the state is "Succeeded", the count is the total number of deleted index records.

Note

  • The delete status will be kept for 14 days. After 14 days it will be purged and you will get a 404 error when querying the deleteID.
  • There is a buffer time of 2 seconds between when data is deleted from the index layer and when it is available to query. E.g., you send a delete request at 01:00:00, then you periodically query the deleteID until 01:02:10 it becomes Succeeded. However, if you query the deleted data at 01:02:11, it may still return. If you query at 01:02:12 or later, the deleted data won't be returned anymore.

results matching ""

    No results matching ""