====== AIRS-2113 ENABLER: Analyze - Handling large single processes (200 MB) ======

===== Common =====

==== Way ====

We can send information about links to upload/download files at least in two different ways:
  * in special XML tags
  * as MIME part of message from type ''text/uri-list''
    * more info: 
      * https://www.iana.org/assignments/media-types/text/uri-list
      * https://tools.ietf.org/html/rfc2483#section-5
      * http://amundsen.com/hypermedia/urilist/


==== Check full chain ====
  * The time to test the "full chain" should be planned.
==== Credentials ====

To download or upload attachments to / from a shared ressource we need credentials. Question: 
  * how to get them?
    * from JBoss (''standalone.xml'' or something like this)? 


==== FAT ====

  * Can FAT work with big files?
    * At least DB column size (max. 2 GB) -> should work, but test it first
  * Maybe it is something already available for sending/receiving links?

==== Antivirus ====
  * Can Antivirus work with big files?
    * Seems to work, because it seems that only the file name (and not payload) is transferred through the SOAP
    * **-** Test on DEV
  * Can the **Antivirus job** work with big files?
    * **-** test it locally
      * what is the job name?
      * com.seeburger.smarti.dao.DAOSysException: Unable to retrieve a database connection: javax.resource.ResourceException: IJ000460: Error checking for a transaction - caused by: com.seeburger.smarti.dao.DAOSysException: Unable to retrieve a database connection: javax.resource.ResourceException: IJ000460: Error checking for a transaction

=== Conclusion ===
  * **[-]** FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte (see ''AttachmentDAOImpl.getFileAttachmentBinary()'') and it takes very long time. For 1GB file my computer needs ca. 50 minutes.

==== Upload ====

  * Is it possible to upload big files?
    * **No**
      * see ''AttachmentUploadHandler:77'' ->  ''this.baos = new ByteArrayOutputStream();'' -> This means the file will be fully loaded into the memory. Which makes working with large files impossible.




===== Import =====
In ''com.seeburger.airsupply.base.exchange.MimeMessageImportHandler'':

<code java>
public void initializeHandlerObjects(SerializableInputStream in, MessageProperties props) {
    super.initializeHandlerObjects(in, props);
    this.mimeMessageImportProcessor = FATProcessorFactory.createMimeMessageImportProcessor(in, props, new XMLImportProviderImpl(),
        AttachmentProcessId.getIdBySmartiMType(props.getProperty(MessageProperties.MESSAGE_TYPE)));
}

</code>

Here we can use different ''mimeMessageImportProcessor'' depending on ''MESSAGE_TYPE''.


===== Misc =====

==== Inserting BLOB for tests ====

To insert BLOB create a new record in FAT META table at first:

<code sql>
INSERT INTO TFATATTACHMENTMETA 
(CMIMECONTENTID,CPROCESSID,CFILENAME,CORIGINATOR,CFILESIZE,CFILETYPE,CENCRYPTED,CDELETEFLAG,CCREATIONTIMESTAMP,CTRANSMISSIONTIMESTAMP,CSCANSTATUS,CSCANMODE,CLASTSCANTIMESTAMP,CWSPROCESSORNODE,CVIRUSIDS,CRETRYBLOCK,CATTACHMENTID,CREADSTATUS,CREADTIMESTAMP,CFLEXFIELD1) 
VALUES 
('VVK_TEST_20190604',6,'attachment_1GB.txt','SUPPLIER',1048576000,'txt',0,0,1555482046306,-9223372036854775808,'STATUS_WATINGFORCHECK','ASYNC',0,null,null,0,3586,'NEW',-9223372036854775808,'COD');
</code>

After that run the small program "Main" from my test project "insertFat" (see ''D:\development\workspaces\insertFat'').


===== Conclusion =====

  * All steps should be checked first
    * To check:
      * JMS as technology
        * **Checked** -> It is not possible. 
          * JMS is not good technology to transfer big data.
          * BIS could not send/receive big files through JMS. (See email by Patrick Bruchhausen from 03.06.2019.)
      * Upload
        * **Checked** -> It is not possible at the moment.
      * FAT
        * **Checked** -> It is not possible at the moment. -> FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte
      * Antivirus 
        * Antivirus job
          * **Checked** -> It is not possible at the moment. -> FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte
        * Antivirus service
          * **TBD**
      * Download
        * **TBD**
      * ...
        * **TBD**
    * The time to test “all steps” should be planned.
  * After that all necessary fixes should be done
    * At least:
      * Replace JMS with other technology
        * Proposal: shared directory
      * FAT 
        * Read/Write files (don't read/write bytewise)
      * Upload
        * Read/write files (don't use ByteBufferedStream)