Amazon S3 Connector

Amazon S3 or Amazon Simple Storage Service is a service offered by Amazon Web Services that provides object storage through a web service interface. Amazon S3 uses the same scalable storage infrastructure that Amazon.com uses to run its global e-commerce network.

Connector Compatibility  

 

 

 

Repo: Yes

Output: Yes

Content Service: Yes

Content Search: No

Manage In Place: No

More info on connection types

Authentication Connection

This connector require an Authentication Connection to function. Authentication connectors are used to authenticate repository/output connections that need certain authentication fields like access tokens or refresh tokens.

  1. Select Authentication from the navigation menu or the dashboard.

  2. Click the New Authentication button

  3. Enter a unique name for the connection and give it a description

  4. Select the connection type from the drop-down and click save.

  5. Fill in the configuration settings on the edit connection page and click save.

Authentication Connections Overview

Authentication connectors are used to authenticate repository/output connections that need certain authentication fields like access tokens or refresh tokens. Click here for more information on setting up authentication connections.

Configuration

  • Name: Unique name for this auth connector.

  • Client ID: The Access Key to connect to the client. For more information about AWS Access Keys, please visit this link.

  • Client Secret: The Secret key associated with the above Access Key.

  • S3 Region: The AWS Region where your instance is located, It will be in the AWS console. default is us-east-1

  • End Point: If using Amazon Glacier, set your instances' url here. It will override the region.

  • Connection Timeout: Set the connection timeout. Higher values may be needed when moving large files.

Tip:  INSTALLED AWS CREDENTIALS
If you leave the Client ID and Client Secret empty, 3Sixty will attempt to authenticate with your installed AWS credentials

Proxy Configuration

This tab is for if you're connecting through a proxy, and is optional. Leave this tab blank if you aren't using a proxy.

  • Proxy User: The proxy user to use. (Optional)

  • Proxy Password: The password for the proxy user. (leave blank if no proxy)

  • Proxy Protocol: The HTTP(S) Protocol to use to connect to the proxy.

  • Full Proxy Url: The Proxy Host (leave blank if no proxy).

  • Proxy Port: The port to connect to on the proxy. (Optional)

  • Proxy Domain: The Domain for the proxy.

  • Proxy Workstation: The workstation to use.

Integration Connection

Every job requires an integration connection for both the source repository connection and the output connection. Also known as input an output connections. Their job is to query or crawl remote systems for files, folders, metadata, versions, and renditions. . In repo mode, it will retrieve list items and all of their relevant metadata from a list or library on the specified site. In output mode, the connection will write content and assign the mapped content type (from type mappings), or simply leave the new list item as a Document. Click here for more information on setting up an integration connection.

  1. Select integration and click on the new connection button.

  2. Enter the name and description of your connection.

  3. Select the connection type from the drop down list.

  4. Click Save on the Create Connection form.

  5. Click Save on the Edit Connection page.

There are no fields to configure in an integration connection.

Guide toIntegration connections

Job Configuration

A 3Sixty Job is the process of moving or syncing content (including versions, ACL's, metadata) from one CMS (content management system) to another. Add tasks to your job to have better control over how your data gets migrated. Click here for details on how to set up an integration job.

Guide to Integration jobs

  1. Select List Jobs under integration on the navigation menu or the dashboard

  2. Click the create job button

  3. In the New Job form

    1. Name the job

    2. Select Simple Migration from the job type drop down

    3. Select the Repository Connection

    4. Select the Output connection

    5. Select your Content Service Connection

      (Only required if you will be using Federation)

    6. Click Save to open the Edit Job page

    7. Fill in the configurations for the Repo and Output Configuration tabs

  4. Click save to save your new integration job

  1. Select Run and Monitor Jobs under Integration in the navigation menu

  2. Click the play button next to the job you want to run

  3. Click the refresh button to view your completed job status

    (Larger jobs will take longer to run)

Repository Specification

Also known as an input connection. It's job is to query or crawl remote systems for files, folders, metadata, versions, and renditions. When using this connector as a source repository filling out the following configuration fields will tell 3Sixty how to locate the files you want migrated.

  • List of S3 Keys: A comma delimited keys of s3 keys (folders) to crawl.

  • Bucket Name: The bucket where the keys are located

  • Metadata is stored is separate files with the suffix .metadata.properties.xml: If this box isn't checked metadata will not be stored.

  • Retrieve File Tags: File tags will be added as metadata with prefix "tag."

  • Max Connections: The maximum number of connections to S3 allowed. Increasing this number will result in faster transfers.

Output Specifications

When using this connector as a file destination, filling out the following fields will tell 3Sixty where you want the files integrated to.

Tip:  There are no actual folders in S3. All files in S3 have a "key", which includes their entire path. The folder path and bucket properties simply prepend these values to each files' keys

  • Output Folder Path: Output folder key. Will be prepended to all document parent paths to make keys.

  • Bucket Name: The bucket name that will be prepended to all keys.

  • Includes Unmapped Properties: Will apply all metadata on the document without mapping

  • Use GZip: Sets whether gzip decompression should be used when receiving HTTP responses.

  • Do not generate XML when Outputting to S3: Like the BFS Connector, the S3 Connector outputs metadata as separate files in the for of [filename].metadata.properties.xml. Check this box if you wish for it to only output files.

  • Use Transfer Manager: If migrating larger files, the S3 APIs offer a transfer manager to ensure more stable uploads

  • Stage Binary to Filesystem: To avoid issues with disconnects from the source, this will temporarily store file content in the Tomcat temp folder before uploading it.

  • Date/DateTime Format: How to format the mapped fields of this type before upload.

Important:  
If migrating large files to S3 it is recommended that you check Use Transfer Manager AND Stage Binary to Filesystem. If you use the Transfer Manager without staging the file, all file uploads will be single threaded by the Transfer Manager.

Advanced Configuration (Output)

  • Max Connections: The maximum number of connections the client can open. Adjusting this can cause changes in performance

  • Multi-value Separator: Some documents have fields that contain multiple values.S3 does not support this, and will use this separator to form a list of these values as a string before upload.

  • Use KMS to encrypt your objects: S3 Buckets use SSE-S3 by default to encrypt objects if you do not specify KMS for encryption.

    • KMS key ARN: If you check the box to use KMS to encrypt your objects you must provide the KMS key ARN.

  • Disable Chunked Encoding: Will remove the transfer-encoding:chunked header from all requests

  • Set Path Style Access: Refer to Amazon's page for more information on this option

  • Object Metadata Fields: A Comma delimited list of fields to add to the S3 Object as User Metadata.

Mapping

3Sixty Mapping gives you the ability to map your content types and metadata from one system to another.

There are two places in 3Sixty where you can set up mappings:

  • At the top of the mapping tab you can select from saved mappings

  • Creating a unique mapping based on your connectors.

Guide to 3Sixty Mapping

Tasks

Job tasks also known as processors provide a processing pipeline for documents and metadata. Post processors can also be added as a task to run after the files are processed.

Click here for a list of all the tasks you can add to your integration job.

  1. Click on the task tab in your integration job.

  2. Find the task you want to create in the drop down. Tasks are grouped by type. You can also search for the task by name.

  3. Select the create task button

  4. Fill out the configuration fields. Some tasks do not require configuration.

  5. Click Done to save the task to your job.

Content Service Connection

This connection will provide a full ECM API for interacting with files, folders, metadata, versions, and renditions.

Content Service Connections Overview

  1. Select Content Service under Connections in the navigation menu

  2. Click the New Connection button

  3. Fill in the following configuration fields.

Basic Configuration

  • Connector ID: Give your connector a unique name

  • Description: Provide a description for this connection

  • Type: Select the Filesystem Content Service Connector

  • Keep Connection Alive: Keep this connection active

  • Keep alive in Milliseconds (300000 is 5 minutes): How long until connection expires if unused

  • Connection URL: The web address for your connection

  • Security Mode: None needed for this connection

  • Mapping Type: Choose single map or group mapping if you are using mapping for jobs

This section covers the S3 specific configuration of the Content Service Connector. For a description of how to set up a content services connector generically see Content Service Connectors.

Configuration

This section covers the S3 specific configuration of the Content Service Connector.

Tip:  S3 DOCUMENT IDS
S3 file ids always take the form of /bucket/(key).

  • Bucket Name: The target bucket for creating a file.

  • Output Folder Path: The key of the folder to target when creating a file.

  • ACL Name: Canned ACL to add to all new content uploaded via this connection.

  • Content Disposition: Default Disposition of any content added via this connection. Will be added to objects metadata

Supported Methods

When using this connector as a content service connector the following methods will be supported.

  • createFile - will take full /bucket/key as folderId parameter to bucket and folder configuration

  • deleteACL

  • deleteObjectByID

  • getACLs

  • getFileContent

  • getObjectProperties

  • listFolderItems

  • lockDocument

  • setACLs

  • unlockDocument

  • updateFile

  • updateProperties

Tip:  S3 ACCESS CONTROL
See this page for information on grantees and permissions.

Get Permissions

GET /api/repo/s3/acls?id=/test-bucket/archive/testdoc.txt

Copy 
{
  "success": true,
  "results": [
    "7cfbdbb50b0682227896f2b416777d4d74906ded4df472db3ace75768962c134:(adminuser):FULL_CONTROL"
  ]
}

Set Permissions

POST /api/repo/s3/acls?id=/test-bucket/archive/testdoc.txt

To add a user to a document, you can use their canonical id or email

Requires a JSON as a request body in the following format:

{"7cfb11150b0682227896f2b416777d4d74906ded4df472db3ace75769062c134":"READ"}

or

{"testuser@gmail.com":"READ"}

which will result in

Copy 
{
  "success": true,
  "results": [
    "7cfbdbb50b0682227896f2b416777d4d74906ded4df472db3ace75768962c134:(adminuser):FULL_CONTROL",
    "7cfb11150b0682227896f2b416777d4d74906ded4df472db3ace75769062c134:(testuser):READ"
  ]
}

To add a group, you'll need the group's URI, such as:

{"http://acs.amazonaws.com/groups/s3/LogDelivery":"WRITE"}

resulting in

Copy 
{
  "success": true,
  "results": [
    "7cfbdbb50b0682227896f2b416777d4d74906ded4df472db3ace75768962c134:(adminuser):FULL_CONTROL",
    "http://acs.amazonaws.com/groups/s3/LogDelivery:(Group):WRITE"
  ]
}

Delete Permissions

Copy 
Delete PermissionsDELETE /api/repo/s3/acls?id=/test-bucket/archive/testdoc.txt&aclId=7cfb11150b0682227896f2b416777d4d74906ded4df472db3ace75769062c134

The aclId parameter can either be the Canonical ID of a user, or the url of the group.

The return will simply be the aclId, but a follow up GET call will produce

Copy 
{
  "success": true,
  "results": [
    "7cfbdbb50b0682227896f2b416777d4d74906ded4df472db3ace75768962c134:(adminuser):FULL_CONTROL"
  ]
}

The items before the semicolon is called the canonical ID of the user. It can be used to remove or update permissions for the user.

Groups use a url instead of a Canonical ID. Such as http://acs.amazonaws.com/groups/global/AllUsers. They will appear as <url>:(Group):<Permission>

API Keys

Amazon S3 Connector: Read=true: Write=true: MIP=false

Repo (Read) Specs

Key

Description

Data Type
crawlBucket Bucket Name String

xml_metadata

Metadata is stored is separate files with the suffix .metadata.properties.xml

Boolean

tags

Retrieve File Tags

Boolean

s3maxconnections

Max Connections

Long

Output (Write) Specs

Key

Description

Data Type
outputfolderpath Output Folder Path String

defaultBucket

Bucket Name

String

includedUnMapped

Include Un-Mapped Properties

Boolean

usegzip

Use GZip

Boolean

file_only

Do not generate XML when Outputting to S3

Boolean

usetm

Use Transfer Manager

Boolean

s3StageToFilesystem

Stage Binary to Filesystem

Boolean

s3dateformat

Date Format

String

s3datetimeformat

Date Time Format

String

s3maxconnections

Max Connections

Long

multivalueseparator

Multi-value field separator

String

kms.encryption

Use KMS to encrypt your objects

Boolean

kms.key.arn

KMS key ARN

String

disablechunkedencoding

Disable Chunked Encoding

Boolean

setpathstyleaccess

Set Path Style Access

Boolean

objectmetadatafields

Object Metadata Fields

String

Need help integrating S3? We can help.