AIRS-2113 ENABLER: Analyze - Handling large single processes (200 MB)

We can send information about links to upload/download files at least in two different ways:

• in special XML tags
• as MIME part of message from type text/uri-list
  • more info:
    • https://www.iana.org/assignments/media-types/text/uri-list
    • https://tools.ietf.org/html/rfc2483#section-5
    • http://amundsen.com/hypermedia/urilist/

• The time to test the “full chain” should be planned.

To download or upload attachments to / from a shared ressource we need credentials. Question:

• how to get them?
  • from JBoss (standalone.xml or something like this)?

• Can FAT work with big files?
  • At least DB column size (max. 2 GB) → should work, but test it first
• Maybe it is something already available for sending/receiving links?

• Can Antivirus work with big files?
  • Seems to work, because it seems that only the file name (and not payload) is transferred through the SOAP
  • - Test on DEV
• Can the Antivirus job work with big files?
  • - test it locally
    • what is the job name?
    • com.seeburger.smarti.dao.DAOSysException: Unable to retrieve a database connection: javax.resource.ResourceException: IJ000460: Error checking for a transaction - caused by: com.seeburger.smarti.dao.DAOSysException: Unable to retrieve a database connection: javax.resource.ResourceException: IJ000460: Error checking for a transaction

• [-] FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte (see AttachmentDAOImpl.getFileAttachmentBinary()) and it takes very long time. For 1GB file my computer needs ca. 50 minutes.

• Is it possible to upload big files?
  • No
    • see AttachmentUploadHandler:77 → this.baos = new ByteArrayOutputStream(); → This means the file will be fully loaded into the memory. Which makes working with large files impossible.

In com.seeburger.airsupply.base.exchange.MimeMessageImportHandler:

public void initializeHandlerObjects(SerializableInputStream in, MessageProperties props) {
    super.initializeHandlerObjects(in, props);
    this.mimeMessageImportProcessor = FATProcessorFactory.createMimeMessageImportProcessor(in, props, new XMLImportProviderImpl(),
        AttachmentProcessId.getIdBySmartiMType(props.getProperty(MessageProperties.MESSAGE_TYPE)));
}

Here we can use different mimeMessageImportProcessor depending on MESSAGE_TYPE.

To insert BLOB create a new record in FAT META table at first:

INSERT INTO TFATATTACHMENTMETA 
(CMIMECONTENTID,CPROCESSID,CFILENAME,CORIGINATOR,CFILESIZE,CFILETYPE,CENCRYPTED,CDELETEFLAG,CCREATIONTIMESTAMP,CTRANSMISSIONTIMESTAMP,CSCANSTATUS,CSCANMODE,CLASTSCANTIMESTAMP,CWSPROCESSORNODE,CVIRUSIDS,CRETRYBLOCK,CATTACHMENTID,CREADSTATUS,CREADTIMESTAMP,CFLEXFIELD1) 
VALUES 
('VVK_TEST_20190604',6,'attachment_1GB.txt','SUPPLIER',1048576000,'txt',0,0,1555482046306,-9223372036854775808,'STATUS_WATINGFORCHECK','ASYNC',0,NULL,NULL,0,3586,'NEW',-9223372036854775808,'COD');

After that run the small program “Main” from my test project “insertFat” (see D:\development\workspaces\insertFat).

• All steps should be checked first
  • To check:
    • JMS as technology
      • Checked → It is not possible.
        • JMS is not good technology to transfer big data.
        • BIS could not send/receive big files through JMS. (See email by Patrick Bruchhausen from 03.06.2019.)
    • Upload
      • Checked → It is not possible at the moment.
    • FAT
      • Checked → It is not possible at the moment. → FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte
    • Antivirus
      • Antivirus job
        • Checked → It is not possible at the moment. → FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte
      • Antivirus service
        • TBD
    • Download
      • TBD
    • …
      • TBD
  • The time to test “all steps” should be planned.
• After that all necessary fixes should be done
  • At least:
    • Replace JMS with other technology
      • Proposal: shared directory
    • FAT
      • Read/Write files (don't read/write bytewise)
    • Upload
      • Read/write files (don't use ByteBufferedStream)