Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 51 additions & 16 deletions sources/platform/integrations/data-and-storage/airbyte.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,35 +7,70 @@ slug: /integrations/airbyte

import ThirdPartyDisclaimer from '@site/sources/_partials/_third-party-integration.mdx';

Airbyte is an open-source data integration platform that allows you to move your data between different sources and destinations using pre-built connectors, which are maintained either by Airbyte itself or by its community.
One of these connectors is the Apify Dataset connector, which makes it simple to move data from Apify datasets to any supported destination.
[Airbyte](https://airbyte.com) is an open-source data integration platform that moves data between sources and destinations using pre-built connectors, maintained by Airbyte or its community. The Apify Dataset connector lets you move data from your Apify datasets to any Airbyte-supported destination.

To use Airbyte's Apify connector you need to:

* Have an Apify account.
* Have an Airbyte account.
This guide shows you how to set up Apify datasets as a source in Airbyte and, optionally, trigger a sync automatically after every Actor run.

<ThirdPartyDisclaimer />

## Set up Apify connector in Airbyte
## Prerequisites

* An [Apify account](https://console.apify.com).
* An [Airbyte account](https://airbyte.com).

Once you have all the necessary accounts set up, you need to set up the Apify connector.
To do so, you will need to navigate to **Sources** tab in Airbyte and select **Apify Dataset**
## Set up the Apify Dataset source in Airbyte

In Airbyte, open the **Sources** tab and select **Apify Dataset**.

![Airbyte sources tab](../images/airbyte-sources.png)

You will need to provide a **dataset ID** and your Apify API Token. You can find both of these in [Apify Console](https://console.apify.com).
Enter your **dataset ID** and **Apify API token**. You can find both in [Apify Console](https://console.apify.com).

![Airbyte source setup](../images/airbyte-source-setup.png)

To find your **dataset ID**, you need to navigate to the **Storage** tab in Apify Console. Copy it and paste it in Airbyte.
To find your **dataset ID**, open the **Storage** tab in Apify Console and copy the ID of the dataset you want to use.

![Datasets in Apify Console](../images/datasets-app.png)

To find your **Apify API token**, open **Settings > API & Integrations** in Apify Console and copy the token.

![API & Integrations token in Apify Console](../images/apify-integrations-token.png)

Your Apify dataset is now available as a source. Next, connect it to a destination so Airbyte knows where to move the data.

## Connect the source to a destination

Setting up the source only tells Airbyte where your data comes from. To actually move it, create a _connection_: an Airbyte pipeline that links your Apify Dataset source to a destination and controls how and when the data syncs. To set one up, follow [Set up a connection](https://docs.airbyte.com/using-airbyte/getting-started/set-up-a-connection) in the Airbyte documentation.

## Trigger a sync automatically after an Actor run

Instead of syncing on a schedule, you can start an Airbyte sync as soon as an Actor run finishes. Use an Apify [webhook](/integrations/webhooks) to call the Airbyte API and refresh the connection whenever new data is scraped.

To set this up, you need an Airbyte connection that uses your Apify Dataset as a source, plus Airbyte API access: an access token and the **connection ID** (copy it from the connection's URL in Airbyte). The exact endpoint and authentication depend on whether you use Airbyte Cloud or a self-managed instance, so check the [Airbyte API reference](https://reference.airbyte.com/reference/createjob) for the current details.

1. In [Apify Console](https://console.apify.com), open the Actor and go to the **Integrations** tab.
1. Under **Connect with Apify**, click **HTTP webhook**.
1. Configure the webhook:
* **Event types**: `Run succeeded` (`ACTOR.RUN.SUCCEEDED`)
* **URL**: the Airbyte API endpoint that starts a sync, for example `https://api.airbyte.com/v1/jobs`
1. Set the **Headers template** to authenticate with your Airbyte token:

![Datasets in app](../images/datasets-app.png)
```json
{
"Authorization": "Bearer YOUR_AIRBYTE_TOKEN",
"Content-Type": "application/json"
}
```

To find your Apify API token, you need to navigate to the **Settings** tab and select **API & Integrations**. Copy it and paste it in the relevant field in Airbyte.
1. Set the **Payload template** to start a sync for your connection:

![Integrations token](../images/apify-integrations-token.png)
```json
{
"connectionId": "YOUR_CONNECTION_ID",
"jobType": "sync"
}
```

And that's it! You now have Apify datasets set up as a Source, and you can use Airbyte to transfer your datasets to one of the available destinations.
1. Click **Save**, then **Test** to confirm that Airbyte starts a sync.

To learn more about how to setup a Connection, visit [Airbyte's documentation](https://docs.airbyte.com/using-airbyte/getting-started/set-up-a-connection)
Every successful Actor run now triggers Airbyte to move the freshly scraped data to your destination, with no manual step in between.
2 changes: 1 addition & 1 deletion sources/platform/integrations/programming/github.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ For private repositories, configure a [deployment key](/actors/development/deplo

## Build automatically on every push

After you link an Actor to a GitHub repository, add a webhook in GitHub to trigger a new build on every push:
By default, a linked Actor rebuilds only when you trigger a build manually. To rebuild it automatically whenever you push to the repository, add a webhook in GitHub that calls the Actor's **Build Actor** API endpoint:

1. In Apify Console, open the Actor's **API** dropdown and select **API endpoints**. Copy the **Build Actor** endpoint URL. It has this format:

Expand Down
Loading