MongoDB GridFS

Note:  INFO
In 3Sixty, the GridFS is just a mode of the MongoDB connector.

GridFS is the MongoDB specification for storing and retrieving large files such as images, audio files, video files, etc. It is kind of a file system to store files but its data is stored within MongoDB collections. GridFS has the capability to store files even greater than its document size limit of 16MB.

GridFS divides a file into chunks and stores each chunk of data in a separate document, each of maximum size 255k.

GridFS by default uses two collections, [collection].files and [collection].chunks to store the file's metadata and the chunks. Each chunk is identified by its unique _id ObjectId field. The [collection].files serves as a parent document. The files _id field in the [collection].chunks document links the chunk to its parent.
More info on MongoDB GridFS

GridFS Authentication Connection

  • Name: The name of your auth connector.

  • Username: The username of the MongoDB admin user you want to authenticate as.

  • Password: The password of the MongoDB admin user you want to authenticate as.

  • Mongo URI: The URI to your MongoDB. For example:mongodb://localhost:27017 will connect to a Mongo database hosted locally (relative to 3Sixty), with the port 27017.

  • Database: The name of the database that you want to authenticate against.

  • Use MongoDB GridFS Services? Required for binary storage. This checkbox enables GridFS services

Note:  MONGO URI
3Sixty inserts the username and password into the connection string. In order to include them as part of the uri we use

[[USER]]:[[PASS]]

Integration Connection

The 3Sixty MongoDB Connector allows organisations to read/write from/toa Mongo Database using GridFS. This means that using 3Sixty and your MongoDB Instance of choice, you can connect to, retrieve data, and content from these instances.

Job Configuration

Output Specification

  • Collection: The name of the collection to create/write to. 3Sixty will handle the "files" and "chunks" collections internally

  • Insert 3Sixty Metadata: Required for Federation. Write 3Sixty metadata onto objects. Automatically included if GridFS is enabled in auth connection.

  • Use bulk write operations (MongoDB only): Ignore for GridFS

  • Number of documents to write per bulk operations: Ignore for GridFS

  • Include Un-Mapped Properties: Add all metadata on the document to the metadata object in [collections].files

Advanced Options

  • Drop and Build Indexes: Indexes are created to speed up searches. This will rebuild them entirely. Should always be checked for the first run for Federation, so you can include a text index for full text search.

  • Index Keys: A comma delimited list of field keys to index. For multiple collections use key1:collection1,key2: collection2.

  • Test Index Keys: Text index keys for full-text searching (comma delimited). Full text search will fail if this is not defined. For multiple collections use key1:collection1,key2:collection2.

  • Upsert Key: 'Upserting' is simply means "Update if exists". This key will be checked to see if the document already exists. Leave blank to use 3Sixty source repository key. If not set and Insert 3Sixty Metadata is not checked, then only creates will be called.

  • MongoDB Write Concern: Write concern describes the level of acknowledgement requested from MongoDB for write operations

  • File Store Connector ID: Connector ID of Content Service Connector to use as the File Store. Or leave blank to use native GridFS. This field is only used if MongoDB GridFS is Enabled in the Authentication Connector.

Repository Configuration

  • Comma delimited list of collections to crawl: Do not append ".files" or ".chunks".

  • Select what field will act as the source id for the document: The field which will appears as the " source_repository_id" field in the output document

  • Query: A Mongo Query. If left blank the query will be "{}", or "get all"

Content Service Connection

Source repository for GridFS are a compound ID take the form of:

[collection]:[mongodbId]:[version]

As an example

demo:61684139dc5eb835dbf0a0c2:1

  • Insert 3Sixty Metadata: Uses 3Sixty specific metadata with objects. File Store Connector ID is only used if this is enabled. Check-in and check-out functionality is only used if this is enabled.

  • GridFS Bucket for file uploads: Name of a single collection

  • File Store Connector ID: Connector ID of Content Service Connector to use as the File Store. Or leave blank to use native GridFS. This field is only used if MongoDB GridFS is Enabled in the Authentication Connector.

  • File Store Root Folder ID: If using a File Store, this value is required. This field is only used if MongoDB GridFS is Enabled in the Authentication Connector.

Supported Methods

  • checkIn

  • checkOut

  • createFile

  • createFolder

  • deleteFolder

  • deleteObjectByID

  • getFileContent

  • getObjectProperties

  • getTypes

  • getVersionContent

  • getVersionProperties

  • listFolderItems

  • listVersions

  • lockDocument

  • unlockDocument

  • updateFile

  • updateProperties

Content Search Connection

Default Query: (3.1.1+) This field allows you to add a default mongodb query to all incoming queries. The query in this box will be wrapped in an $and clause with all other search parameters. Adding the "metadata" prefix is required for all fields, except the following

  • length

  • filename

  • chunkSize

  • uploadDate

  • md5

To add a default query which filters out all .txt documents:

Copy 
{
  "metadata.simflofy_content_type": {
    "$not": {
      "$eq": "text/plain"
    }
  }
}

Get all versions and display versions on file name: As the name says, search will now retrieve all versions of documents and label them