Salesforce Analytics Cloud Connector

[tab,title="Studio Visual Editor"]
....

. Create a new Mule Project by clicking on *File* > *New* > *Mule Project*. In the new project dialog box, the only thing you are required to enter is the name of the project. Click on *Finish*.
+
image:new_project_dialog.png[New project dialog]
+
. Now let's create the flow. Navigate through the project's structure and double-click on *src/main/app/project-name.xml* and follow the steps below.
+
. On the right side of studio search for *File*.
+
image:search_for_file.png[Search for File]
. Drag the *File* element onto the canvas.
. Search for *DataMapper* and drag it after *File*.
. Search for *Salesforce Analytics Cloud* and drag it after *DataMapper*.
. After completing the previous steps you should see:
+
image:all_flow_unconfigured.png[Unconfigured All In One flow]
. Let's start configuring each element. Double-click on the *File* element.
+
image:file_component.jpg[File component]
. Click on `...` next to the *Path* field.
. Choose a folder with only the csv file that you want to upload. You can download our example file and save it into chosen folder.
+
link:_attachments/CsvDemoTestData.csv[CsvDemoTestData.csv]
. Double-click on *Salesforce Analytics Cloud* connector.
. Click on the plus sign next to the *Connector configuration* dropdown.
+
image:create_data_set_config.jpg[Create data set config]
. A pop-up appears asking for type of configuration. Choose *Salesforce Analytics Cloud: Basic Authentication* option and click *OK*.
. A new pop-up appears asking for information required for basic authentication. For more info see the <<Installing and Configuring, Installing and Configuring>> section
+
image:basic_authentication_config.png[Basic Auth config]
. In the *Connection* section enter the credentials used to access the Salesforce instance.
. In the *DataSense metadata* section for the *Metadata file name* field enter the filename that describes the data structure you are going to upload. The filename has to be relative to the *src/main/resources* directory of your Studio project. For the file provided a few steps earlier (CsvDemoTestData.csv) you can use the metadata file provided below but do not forget to copy it into the *src/main/resources* directory.
+
link:_attachments/metadata.json[metadata.json]
. Click *OK* to return to the Salesforce Analytics Cloud tab.
. From the *Operation* dropdown in the *Basic Settings* section choose *Upload external data into new data set and start processing*.
. From the *Operation* dropdown in the *DataSet info* section choose *OVERWRITE*.
. In the *Description* enter *Test data set*.
. In the *Label* field under *DataSet info* enter *Test data set*.
. In the *Name* field under *DataSet info* enter *test_data_set*.
. Double-click on the *DataMapper* element.
. Click on the *Type* dropdown in the Input section and choose *CSV*
+
image:csv_list_record.png[Data Mapper set CSV as input type]
. Click on `...` next to the *CSV* field of the Input section and browse to the csv file in the same folder you selected for the *File* connector.
. Click the *Create mapping* button and you should see something similar to the picture below.
+
image:DM_mappings.png[Data mapper mappings]
. Now everything is set up and the application can be deployed.

It's time to test the app. Run the app in Anypoint Studio (Right-click on project name > *Run as > Mule Application*). Monitor the studio console and check Salesforce Wave Analytics UI to see if the data was uploaded.
....
[tab,title="XML Editor"]
....
. Add the sfdc-analytics namespace to the mule element as follows:

+

[source,xml,linenums]
----
xmlns:sfdc-analytics="http://www.mulesoft.org/schema/mule/sfdc-analytics"
----

. Add the location of the analytics schema referred to by the sfdc-analytics namespace:

+

[source,xml,linenums]
----
http://www.mulesoft.org/schema/mule/sfdc-analytics http://www.mulesoft.org/schema/mule/sfdc-analytics/current/mule-sfdc-analytics.xsd
----

. Add the data-mapper namespace as follows:

+

[source,xml,linenums]
----
xmlns:data-mapper="http://www.mulesoft.org/schema/mule/ee/data-mapper"
----

. Add location of data mapper schema referred by data-mapper namespace with the following value:

+

[source,xml,linenums]
----
http://www.mulesoft.org/schema/mule/ee/data-mapper http://www.mulesoft.org/schema/mule/ee/data-mapper/current/mule-data-mapper.xsd
----

. Add a context:property-placeholder element to your project, then configure its attributes as follows:

+

[source,xml]
----
<context:property-placeholder location="mule-app.properties"/>
----

. Add a data-mapper:config element to your project, then configure its attributes as follows:

+

[source,xml,linenums]
----
<data-mapper:config name="CSV_To_List_Record_" transformationGraphPath="csv_to_list_record_.grf" doc:name="CSV_To_List_Record_"/>
----

. Add a sfdc-analytics:config element to your project, then configure its attributes as follows:

+

[source,xml,linenums]
----
<sfdc-analytics:config name="Salesforce_Analytics_Cloud__Basic_authentication" username="${salesforce.username}" password="${salesforce.password}" securityToken="${salesforce.securityToken}" metadataFileName="${metadata.file.analytics}" doc:name="Salesforce Analytics Cloud: Basic authentication" url="${salesforce.url}"/>
----

. Add an empty flow element to your project as follows:

+

[source,xml,linenums]
----
<flow name="analytics_performanceFlow">
</flow>
----

. Within the flow element add a file:inbound-endpoint element as follows:

+

[source,xml,linenums]
----
<file:inbound-endpoint path="path_to_folder_to_monitor" moveToDirectory="path_to_folder_where_to_move_processed_files" responseTimeout="10000" doc:name="File">
</file:inbound-endpoint>
----

. Within the flow element add a data-mapper:transform element as follows:

+

[source,xml,linenums]
----
<data-mapper:transform config-ref="CSV_To_List_Record_" doc:name="CSV To List&lt;Record&gt;"/>
----

. Within the flow element add a sfdc-analytics:upload-external-data-into-new-data-set-and-start-processing element as follows:

+

[source,xml,linenums]
----
<sfdc-analytics:upload-external-data-into-new-data-set-and-start-processing config-ref="Salesforce_Analytics_Cloud__Basic_authentication1" type="recordId" operation="UPSERT" description="Test upload of 2500 records all in one step" label="records_2500_in_one_step" dataSetName="records_2500_in_one_step_with_app" edgemartContainer="TestContainer" notificationSent="ALWAYS" notificationEmail="name@email.com" doc:name="Salesforce Analytics Cloud">
    <sfdc-analytics:payload ref="#[payload]"/>
</sfdc-analytics:upload-external-data-into-new-data-set-and-start-processing>
----

. In the end the xml file should look like this:

+

[source,xml,linenums]
----
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:file="http://www.mulesoft.org/schema/mule/file"
	xmlns:context="http://www.springframework.org/schema/context"
	xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
	xmlns:spring="http://www.springframework.org/schema/beans"
	xmlns:sfdc-analytics="http://www.mulesoft.org/schema/mule/sfdc-analytics"
	xmlns:data-mapper="http://www.mulesoft.org/schema/mule/ee/data-mapper"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.mulesoft.org/schema/mule/sfdc-analytics http://www.mulesoft.org/schema/mule/sfdc-analytics/current/mule-sfdc-analytics.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/ee/data-mapper http://www.mulesoft.org/schema/mule/ee/data-mapper/current/mule-data-mapper.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-current.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd">
	<context:property-placeholder location="mule-app.properties"/>
	<sfdc-analytics:config name="Salesforce_Analytics_Cloud__Basic_authentication" username="${salesforce.username}" password="${salesforce.password}" securityToken="${salesforce.securityToken}" metadataFileName="${metadata.file.analytics}" doc:name="Salesforce Analytics Cloud: Basic authentication" url="${salesforce.url}"/>
	<data-mapper:config name="CSV_To_List_Record_" transformationGraphPath="csv_to_list_record_.grf" doc:name="CSV_To_List_Record_"/>
	<flow name="analytics_performanceFlow">
        <file:inbound-endpoint path="path_to_folder_to_monitor" moveToDirectory="path_to_folder_where_to_move_processed_files" responseTimeout="10000" doc:name="File">
        </file:inbound-endpoint>
        <data-mapper:transform config-ref="CSV_To_List_Record_" doc:name="CSV To List&lt;Record&gt;"/>
        <sfdc-analytics:upload-external-data-into-new-data-set-and-start-processing config-ref="Salesforce_Analytics_Cloud__Basic_authentication" type="recordId" operation="UPSERT" description="Test upload of 2500 records all in one step" label="records_2500_in_one_step" dataSetName="records_2500_in_one_step_with_app" edgemartContainer="TestContainer" notificationSent="ALWAYS" notificationEmail="name@email.com" doc:name="Salesforce Analytics Cloud">
            <sfdc-analytics:payload ref="#[payload]"/>
        </sfdc-analytics:upload-external-data-into-new-data-set-and-start-processing>
    </flow>
</mule>
----
....

[tab,title="Studio Visual Editor"]
....
. Create a new Mule Project by clicking on *File* > *New* > *Mule Project*. In the new project dialog box, the only thing you are required to enter is the name of the project. Click *Finish*.
+
image:new_project_dialog.png[New project dialog]
+
. Now let's create the flow. Navigate through the project's structure and double click on *src/main/app/project-name.xml* and follow the steps below.
. On the right side of Studio search for *Batch*.
+
image:search_for_batch.jpg[Search for batch]
. Select *Batch* and drag it onto the canvas.
+
image:batch_component.jpg[Batch component on canvas]
+
[NOTE]
When using the *Batch* component, tune it based on the amount of memory that you provide to the Mule server.
+
. Similar to what was done in step 1, search for *File*.
. Drag *File* into the *Input* section of the batch element created earlier.
. Search for *Message Enricher*, then drag and drop it after *File*.
. Search for *DataMapper* and drag it after *Message Enricher*.
. Search for the connector named *Salesforce Analytics Cloud* and drag it into *Message Enricher*.
. Search for the *Batch Commit* component in the palette and drag it into the *Batch step* section of *Batch*.
+
[NOTE]
When using *DataMapper*, ensure the "Streaming" option in the *Batch Commit* component is enabled. This way you avoid loading the entire input in memory.
+
. Search for the connector named *Salesforce Analytics Cloud* and drag it into the *Batch Commit* section of *Batch step*.
+
[NOTE]
====
Bear in mind that the default threading profile uses 16 threads and each thread is loading data in chunks of 100 records until it reaches the "Commit size" set on the *Batch Commit* component.
You can minimize the memory used by decreasing the number of threads.

Finally you have to be aware of the fact that Salesforce Analytics Cloud connector is also using some memory internally and you should tune the "Commit Size" in the *Batch Commit* component for the purpose of efficiency, therefore do not set the parameter too low.
====
+
. Drag another *Salesforce Analytics Cloud* connector into the *On complete* section of *Batch*.
. After completing all the above steps you should see:
+
image:batch_flow_unconfigured.png[Unconfigured Batch flow]
. Lets start configuring each element. Double click on the *File* element.
+
image:file_component.jpg[File component]
. Click on `...` next to the *Path* field.
. Choose a folder with only the csv file that you want to upload. You can download our example file and save it into your chosen folder.
+
link:_attachments/CsvDemoTestData.csv[CsvDemoTestData.csv]
. Double-click on the *Salesforce Analytics Cloud* connector in the *Message Enricher*.
. Click on the plus sign next to the *Connector configuration* dropdown.
+
image:create_data_set_config.jpg[Create data set config]
. A pop-up asking for type of configuration appears. Choose the *Salesforce Analytics Cloud: Basic Authentication* option and click *OK*.
. A new pop-up asks for information required for basic authentication. For more info see the <<Installing and Configuring, Installing and Configuring>> section
+
image:basic_authentication_config.png[Basic Auth config]
. In the *Connection* section enter the credentials used to access the Salesforce instance.
. In the *DataSense metadata* section for the *Metadata file name* field enter the filename that describes the data structure you are going to upload. The filename has to be relative to the *src/main/resources* directory of your Studio project. For the file provided a few steps earlier (CsvDemoTestData.csv) you can use the metadata file provided below, but do not forget to copy it into the *src/main/resources* directory.
+
link:_attachments/metadata.json[metadata.json]
. Click *OK* to return to the Salesforce Analytics Cloud tab.
. From the *Operation* dropdown in the *Basic Settings* section choose *Create data set*.
. From the *Operation* dropdown in the *DataSet info* section choose *OVERWRITE*.
. In the *Description* field enter *Test data set*.
. In the *Label* field under *DataSet info* enter *Test data set*.
. In the *Name* field under *DataSet info* enter *test_data_set*.
. Double-click on *Message Enricher* and fill in the fields as below.
+
image:message_enricher_config.jpg[Message Enricher Config]
. Double-click on *Batch Commit* from *Batch step*.
. For *Commit size* enter the number of records you want to process in one step. (e.g. 5000)
+
[NOTE]
The application is logging a warning message if the data provided for processing in one step exceeds the size of data accepted by Analytics Cloud System in one step. The message looks like this: "The size of data provided for processing in one step exceeded the maximum size of one chunk allowed by Analytics Cloud System. In order to optimize the memory used you should decrease the size of data provided in one step. If you see this message then you should tune the *Commit Size* by decreasing it until you do not see the message anymore.
+
. Double-click on *Salesforce Analytics Cloud* from *Batch Commit*.
. From the *Connector configuration* dropdown choose *Salesforce_Analytics_Cloud__Basic_authentication* (only this option should be available).
. Choose *Upload external data* as the operation.
. Check the bottom corner on the right-hand side and wait for DataSense to fetch metadata.
+
image:fetch_metadata_bar.jpg[Fetch metadata progress bar]
. For *Data Set Id* enter *#[variable:dataSetId]*
. Double-click on the *DataMapper* element.
. Click on the *Type* dropdown in the Input section and choose *CSV*
+
image:csv_list_record.png[Data Mapper set CSV as input type]
. Click on `...` next to the *CSV* field of the Input section and browse to the csv file in the same folder you selected for the *File* connector.
. Click the *Create mapping* button and you should see something like the following.
+
image:DM_mappings.png[Data mapper mappings]
. Double-click on *Salesforce Analytics Cloud* from the *On complete* section of *Batch*.
. From the *Connector configuration* dropdown select *Salesforce_Analytics_Cloud__Basic_authentication* (only this option should be available).
. From the *Operation* dropdown select *Start data processing*.
. In the *Data Set Id* field enter `#[variable:dataSetId]`
. At this point, everything should be set up and the application can be deployed.

It is time to test the application. Run the application in Anypoint Studio (Right click on the project name in Studio's package explorer and select *Run as -> Mule Application*). Monitor the studio console and check Salesforce Wave Analytics UI to see if the data was uploaded.
....
[tab,title="XML Editor"]
....

. Add sfdc-analytics namespace to mule element as follows:

+

[source,xml,linenums]
----
xmlns:sfdc-analytics="http://www.mulesoft.org/schema/mule/sfdc-analytics"
----

. Add location of analytics schema referred by sfdc-analytics namespace with the following value:

+

[source,xml,linenums]
----
http://www.mulesoft.org/schema/mule/sfdc-analytics http://www.mulesoft.org/schema/mule/sfdc-analytics/current/mule-sfdc-analytics.xsd
----

. Add data-mapper namespace as follows:

+

[source,xml]
----
xmlns:data-mapper="http://www.mulesoft.org/schema/mule/ee/data-mapper"
----

. Add location of data mapper schema referred by data-mapper namespace with the following value:

+

[source,xml,linenums]
----
http://www.mulesoft.org/schema/mule/ee/data-mapper http://www.mulesoft.org/schema/mule/ee/data-mapper/current/mule-data-mapper.xsd
----

. Add a context:property-placeholder element to your project, then configure its attributes as follows:

+

[source,xml]
----
<context:property-placeholder location="mule-app.properties"/>
----

. Add a data-mapper:config element to your project, then configure its attributes as follows:

+

[source,xml,linenums]
----
<data-mapper:config name="CSV_To_List_Record_" transformationGraphPath="csv_to_list_record_.grf" doc:name="CSV_To_List_Record_"/>
----

. Add a sfdc-analytics:config element to your project, then configure its attributes as follows:

+

[source,xml,linenums]
----
<sfdc-analytics:config name="Salesforce_Analytics_Cloud__Basic_authentication" username="${salesforce.username}" password="${salesforce.password}" securityToken="${salesforce.securityToken}" metadataFileName="${metadata.file.analytics}" doc:name="Salesforce Analytics Cloud: Basic authentication" url="${salesforce.url}"/>
----

. Add an empty batch:job element to your project as follows:

+

[source,xml,linenums]
----
<batch:job name="demoBatch">
    <batch:input>
    </batch:input>
    <batch:process-records>
    </batch:process-records>
    <batch:on-complete>
    </batch:on-complete>
</batch:job>
----

. Add a file:inbound-endpoint element into batch:input of batch:job, then configure it as follows:

+

[source,xml,linenums]
----
<file:inbound-endpoint path="path_to_folder_to_monitor" moveToDirectory="path_to_folder_where_to_move_processed_files" responseTimeout="10000"
                       doc:name="File For Batch">
</file:inbound-endpoint>
----

. Add an empty enricher element into batch:input of batch:job, then configure it as follows:

+

[source,xml,linenums]
----
<enricher source="#[payload]" target="#[variable:dataSetId]" doc:name="Message Enricher">
</enricher>
----

. Add a sfdc-analytics:create-data-set element into enricher, then configure it as follows:

+

[source,xml,linenums]
----
<sfdc-analytics:create-data-set config-ref="Salesforce_Analytics_Cloud__Basic_authentication" operation="OVERWRITE" description="${batch.dataSetDescription}" label="${batch.dataSetLabel}" dataSetName="${batch.dataSetName}" edgemartContainer="${batch.dataSetEdgemartContainer}" notificationSent="ALWAYS" notificationEmail="name@email.com" doc:name="Salesforce Analytics Cloud"/>
----

. Add a data-mapper:transform element into batch:input of batch:job, then configure it as follows:

+

[source,xml,linenums]
----
<data-mapper:transform config-ref="CSV_To_List_Record_" doc:name="CSV To List&lt;Record&gt;"/>
----

. Add an empty batch:step element into batch:process-records of batch:job, then configure it as follows:

+

[source,xml,linenums]
----
<batch:step name="Batch_Step">
</batch:step>
----

. Add an empty batch:commit element into batch:step of batch:process-records, then configure it as follows:

+

[source,xml,linenums]
----
<batch:commit  doc:name="Batch Commit" size="3000">
</batch:commit>
----

. Add a sfdc-analytics:upload-external-data element into batch:commit of batch:step of batch:process-records, then configure it as follows:

+

[source,xml,linenums]
----
<sfdc-analytics:upload-external-data config-ref="Salesforce_Analytics_Cloud__Basic_authentication" type="recordId" dataSetId="#[variable:dataSetId]" doc:name="Salesforce Analytics Cloud">
    <sfdc-analytics:payload ref="#[payload]"/>
</sfdc-analytics:upload-external-data>
----

. Add a sfdc-analytics:start-data-processing element into batch:on-complete of batch:job, then configure it as follows:

+

[source,xml,linenums]
----
<sfdc-analytics:start-data-processing config-ref="Salesforce_Analytics_Cloud__Basic_authentication" dataSetId="#[variable:dataSetId]" doc:name="Salesforce Analytics Cloud"/>
----

. In the end the xml file should look like this:

+

[source,xml,linenums]
----
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:batch="http://www.mulesoft.org/schema/mule/batch"
	xmlns:file="http://www.mulesoft.org/schema/mule/file"
	xmlns:context="http://www.springframework.org/schema/context"
	xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
	xmlns:spring="http://www.springframework.org/schema/beans"
	xmlns:sfdc-analytics="http://www.mulesoft.org/schema/mule/sfdc-analytics"
	xmlns:data-mapper="http://www.mulesoft.org/schema/mule/ee/data-mapper"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/batch http://www.mulesoft.org/schema/mule/batch/current/mule-batch.xsd
http://www.mulesoft.org/schema/mule/sfdc-analytics http://www.mulesoft.org/schema/mule/sfdc-analytics/current/mule-sfdc-analytics.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/ee/data-mapper http://www.mulesoft.org/schema/mule/ee/data-mapper/current/mule-data-mapper.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-current.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd">
	<context:property-placeholder location="mule-app.properties"/>
	<sfdc-analytics:config name="Salesforce_Analytics_Cloud__Basic_authentication" username="${salesforce.username}" password="${salesforce.password}" securityToken="${salesforce.securityToken}" metadataFileName="${metadata.file.analytics}" doc:name="Salesforce Analytics Cloud: Basic authentication" url="${salesforce.url}"/>
	<data-mapper:config name="CSV_To_List_Record_" transformationGraphPath="csv_to_list_record_.grf" doc:name="CSV_To_List_Record_"/>
	<batch:job name="demoBatch">
        <batch:input>
            <file:inbound-endpoint path="path_to_folder_to_monitor" moveToDirectory="path_to_folder_where_to_move_processed_files" responseTimeout="10000"
                                   doc:name="File For Batch">
            </file:inbound-endpoint>
            <enricher source="#[payload]" target="#[variable:dataSetId]" doc:name="Message Enricher">
                <sfdc-analytics:create-data-set config-ref="Salesforce_Analytics_Cloud__Basic_authentication" operation="OVERWRITE" description="${batch.dataSetDescription}" label="${batch.dataSetLabel}" dataSetName="${batch.dataSetName}" edgemartContainer="${batch.dataSetEdgemartContainer}" notificationSent="ALWAYS" notificationEmail="name@email.com" doc:name="Salesforce Analytics Cloud"/>
            </enricher>
            <data-mapper:transform config-ref="CSV_To_List_Record_" doc:name="CSV To List&lt;Record&gt;"/>
        </batch:input>
        <batch:process-records>
            <batch:step name="Batch_Step">
                <batch:commit  doc:name="Batch Commit" size="3000">
                    <sfdc-analytics:upload-external-data config-ref="Salesforce_Analytics_Cloud__Basic_authentication" type="recordId" dataSetId="#[variable:dataSetId]" doc:name="Salesforce Analytics Cloud">
                        <sfdc-analytics:payload ref="#[payload]"/>
                    </sfdc-analytics:upload-external-data>
                </batch:commit>
            </batch:step>
        </batch:process-records>
        <batch:on-complete>
            <sfdc-analytics:start-data-processing config-ref="Salesforce_Analytics_Cloud__Basic_authentication" dataSetId="#[variable:dataSetId]" doc:name="Salesforce Analytics Cloud"/>
        </batch:on-complete>
    </batch:job>
</mule>
----
....