Manage In Place

Purpose

Manage in Place (MIP) is intended for organisations who have their content spread out across multiple repositories (file shares, M365 sources like SharePoint, Exchange, OneDrive, or other line of business applications) and wish to centrally govern the content while meeting the records compliance needs. It allows users to manage their content's lifecycle through Objective ECM without the need to transfer it from its original location.

How it works

Objective 3Sixty will read documents from a source system and extract various metadata from them. The standard metadata to find the document in the source system is always included. Using this metadata it will construct a record “stub” in the target system (in this case Objective ECM/Nexus). That stub will allow users to interact with the document in Objective ECM through Content Services. Interactions with the document will be managed by 3Sixty.

 

In Manage In Place users can:

  • View the source system metadata in Objective ECM/Nexus directly

  • You will be able to search using the captured metadata

  • You can run a workflow on these captured record stubs

  • You will be able to leverage your existing Objective ECM/Nexus disposal policies to destroy content from the source repositories

Manage In Place Setup

Read

In order to set up MIP, you must first have a working connection to the source system. This process will be highly dependent on what system you’re attempting to read from. This information is available in our documentation for the repositories we support. This will affect the job read options in your job.
3Sixty Connectors

Content Services

A content service connection for the system managed is required for a MIP job. It can (and usually should) use the same authentication connection. Some content service connections have additional configuration, found in the Configurations Parameters tab.
Content Service Connections

Output

You will need a working connection to Objective ECM/Nexus. For Objective ECM, you can authenticate either using the Objective Authentication Connector for basic (username/password) authentication or an Objective OAuth Connector. Objective Nexus does not support basic authentication and only OAuth authentication is supported.

Important:  An ECM Version of 11.3 or later must be used for Manage In place.


Objective Connector

Job

In the Integration > List Jobs menu,

1. Click the large circle with a plus sign

2. Name the job (unique name required)

3. Select “Manage in Place” as the job type.

3. Select the Repository Connection for the repository to be managed

4. Select the Output Connection for your ECM/Nexus instance. You may see a Reporting connection available, ignore it.

5. Select the Content Service Connection for the repository to be managed

6. (Optional) Create a job group for your MIP job.

The MIP Job Type and Configuration

MIP jobs are a type of job known as Incremental Migration. In Objective 3Sixty, when reading from a repository, the last modified date of the document is checked against a time range to determine whether the document should be included in the run. For MIP-type jobs, this value is automatically calculated using the following rules:

If the job has never been run before, it will use January 1st 1970 as its “start time” and the current time as the “end time”. It will likely pick up all documents in the configured target directories. The documents processed can be limited by tasks

If a job has been run previously, the “start time” will be the moment the that last job started. This means it will pick up all documents modified since the last time the job was started. The “end time” is always the current date and time.

Note:  START AND END TIME
These values are only set by users for Simple Migrations. For Incremental Migrations, they are calculated based on when the job was last run. Event and Polling jobs don't use them at all

Details Tab:

Name: Descriptive Name of the Job that will be displayed throughout the UI.

Repository Connection: The Repository Connection used for this Job. Options are limited by job type

Output Connection: The Output Connection used for this Job. Options are limited by job type

Content Service Connector: A content service connection for the repository to be read. If not set, the job will not start.

Type: The type of job. In this case Manage in Place. Changing this will reload the page with appropriate connectors for the type of job selected.

Include File Content: When 3Sixty reads documents, bringing in the content is optional - as we are creating a record stub in the destination system and not copying the content. Leave this checked if you want to analyse the content for detecting duplicates or personally identifiable information. Manage in Place jobs can read the content but will not write anything by metadata.

Job Tags: Another option for separating jobs into groups. If any jobs have tags, they will appear as a filtering option when listing jobs.

Date/Time Options
In job advanced options, when keeping both the repository time zone and output time zone in Australia Sydney time, uploading new documents/updating existing documents, and running the manage in place job, it will pick up the newly uploaded documents or updated documents

Caution:  The time for the modified date value on ECM will not following Australia time. This may be caused by the SharePoint time zone. If changing the repository time zone the same as the SharePoint time zone(New York time), the newly uploaded documents/updated documents will not be picked up since the time is earlier than the last time run.

Mapping Tab:

Pre-requisite considerations

In ECM/Nexus, make sure that the appropriate "External Record" object is created. This object may be a sub-type of the "Document" object. Users can add the following metadata fields to your "External Record" object to provide more information about the document: Creating Fields in ECM

  • Governance External ID

  • Governance Created By

  • Governance Created Date

  • Governance Modified By

  • Governance Modified Date

  • Governance External Version

  • Governance URL

If you want any other custom metadata to be added to your record stub, make sure to include them in your External record catalogue.

In 3Sixty, Create a Job Mapping for ECM/Nexus.

  • Navigate to Integration -> Job Mappings

  • Click on the + button to create a new job mapping

  • Give a name to your mapping

  • If you have run discovery schema on the repositories, you will be able to use them to specify the mapping. You will be able to know the source and target field names/IDs by running Discovery schema on the corresponding sources. Visit the <<discovery schema>> page to learn more about how to discover the schema of your source repository.

  • Map corresponding "External Record" object ID to document object in 3Sixty

  • Map the corresponding catalogue ID to document aspect mapping in 3Sixty

  • Map individual metadata field ID/text internal names to the corresponding source document fields in 3Sixty

  • If you want to enable auditing of any of these metadata fields, make sure to click on the "eye" icon to enable auditing.

  • Click on "Save" to save your job mapping

Mapping tab in Job configuration

  • In the mapping tab, expand the "Add additional job mappings or mapping groups" section.

  • Select the mapping job that you created above

  • Alternatively, you can also do the field mappings directly in the job but that mapping can not be used in other jobs, so it is recommended to create job mappings if you need to reuse the mapping in other jobs as well.

Note:  Email types do not get the Metadata when using Manage In Place.

Auditing

Audit: Where to write audit data too. Note that Job Restart and Document Search will only work if Mongo is selected.

Mongo: Audit to the Mongo Database or SQL Database if running in SQL mode.

Log4j: Audit to the 3sixty-admin.log

None: Auditing is turned off.

Record Audit When: The audit level or what to audit.

All (Read, Written, Skipped, Deleted and Failed): Full Auditing

Failures Only: Audit only those records that fail during a job run.

Failed and Deleted: Audit a record that failed during the job run or was deleted from a Sync Job.

Failed and Skipped: Audit a record that failed during the job run or was skipped.

Failed, Deleted and Written: Audit a record that failed during the Job run or was deleted from a Sync Job or written by the Output Connection.

Audit All Mappings: (Advanced usage) Mapping audits can be used to see how certain fields are being mapped. Checking this will create audits for all mappings on every run document. Use with caution. Individual mapping audits. can be set in the Mappings tab.

Notifications:

Notifications will only function if email is enabled. This allows the user to set conditions in which the listed email addresses will receive confirmed status reports when a job completes.

Tasks

Tasks allow individual actions to be run on each document picked up during the read. If you wish to perform additional analysis or filtering, it can be done in tasks. A full list of tasks can be found here. We’ll focus on the additional data extraction tasks mentioned previously. Tasks are run in the order they are set in the list, from top to bottom. Use default values unless otherwise noted.

Apache Tika is an open-source tool used to extract text from documents.

  • Tika Content Field: The field where the task will put the extracted content.

  • Max Content Length (B): Content length, checked before processing. Only extract from documents with length under this value. Set to 0 to process documents of any length. The Tika library imposes a 100,000-character limit on documents and will throw an error if a document larger than that is extracted. This cannot currently be changed.

  • File Extensions to Extract: Comma delimited list of extensions. Extensions are checked at the same time as the content length

  • Content Length to Retrieve (B): The amount of document content to extract. -1 for the entire document.

  • Fail on Extract: If the extraction fails, fail the document.

  • Remove Binary: Remove the binary from the documents. Will happen even if document exceeds maximum length.

    Uncheck this if the pii detection task or duplicate detection task comes after it or the documents will fail those tasks, since the content has been removed.

  • Stage binary on filesystem: For lower memory systems, stages to the filesystem instead of in memory.

The task will extract a document’s content and run a series of preconfigured Regular Expressions to check for the presence of specific content. These rules are configurable by an Admin in the Admin > PII Properties menu.

The task will add the metadata flag hasPii to each document for the purpose of further analysis and filtering.

Field To Mark:

  • The field where the map of PII will be stored.

Fields To Check:

  • Source properties and/or document to check for PII. Use ALL_PROPS to check all properties, BINARY to check the document (extracted via Tika) or individual property names.

  • Break up PII data into individual fields with a prefix (Ex. prefix_CreditCard):

    This checkbox is recommended for easier mapping of PII data. If left unchecked, the data will be stored in a more difficult-to-parse format

A hash value is a sort of fingerprint for a document. The entirety of the document’s content is processed into a single series of characters which can be compared to other hashes to check for duplicate data. This task is required for duplicate detection and must always come before it in a task list

  • Staging: Whether the document hash is created by reading the document directly from its source, or whether it's staged in the filesystem first. Staging to the filesystem is useful for systems that shut down their connections without warning.

  • Algorithm: (Advanced) picked which type of cryptographic processing is done on the document.

This task is used to check historical data that has passed through 3Sixty to see if duplicates are being read.

  • Field to Compare: The field whose value will be used to check for duplicates. If this value is found in any other document it will be considered a duplicate. Default is the file hash

  • Duplication Check Scope: This is how deeply 3Sixty will search its audits for previously run versions of the document. There are three options:

  • Job: Every document ever run by the specific job using the task

  • Enterprise: Every document ever run by the 3Sixty instance.

  • Job Run Only: Useful for testing. Will check documents previously audited as part of the current job run. Ie, it will only detect a duplicate if it's being run in the same job at the same time. Highly recommended to set it to "Job Run Only" when you are testing or figuring out your configurations. This needs to be changed to Job or Enterprise on production use after your jobs are configured correctly.

  • Action: What to do if a duplicate is found

  • Audit and Continue: The document will be audited as a duplicate but otherwise continue to the next task or output

  • Skip: The document will be audited as skipped and will not go on to the next task, or output

  • Fail: The document will be audited as a failure and will not go on to the next task, or output

Important:  Schedule your MiP job to run on a regular basis.

Handling of external records in ECM/Nexus

Event Audit Log

 

Related Articles

PII Detection

Jobs Overview

Duplication Check