vwiki

User Tools

Site Tools

Trace:
airs-2113

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
airs-2113 [2019/06/04 12:25] – [Conclusion] kalaschnikowairs-2113 [2019/06/04 13:06] (current) – [Conclusion] kalaschnikow
Line 1: Line 1:
 +====== AIRS-2113 ENABLER: Analyze - Handling large single processes (200 MB) ======
 +
 +===== Common =====
 +
 +==== Way ====
 +
 +We can send information about links to upload/download files at least in two different ways:
 +  * in special XML tags
 +  * as MIME part of message from type ''text/uri-list''
 +    * more info: 
 +      * https://www.iana.org/assignments/media-types/text/uri-list
 +      * https://tools.ietf.org/html/rfc2483#section-5
 +      * http://amundsen.com/hypermedia/urilist/
 +
 +
 +==== Check full chain ====
 +  * The time to test the "full chain" should be planned.
 +==== Credentials ====
 +
 +To download or upload attachments to / from a shared ressource we need credentials. Question: 
 +  * how to get them?
 +    * from JBoss (''standalone.xml'' or something like this)? 
 +
 +
 +==== FAT ====
 +
 +  * Can FAT work with big files?
 +    * At least DB column size (max. 2 GB) -> should work, but test it first
 +  * Maybe it is something already available for sending/receiving links?
 +
 +==== Antivirus ====
 +  * Can Antivirus work with big files?
 +    * Seems to work, because it seems that only the file name (and not payload) is transferred through the SOAP
 +    * **-** Test on DEV
 +  * Can the **Antivirus job** work with big files?
 +    * **-** test it locally
 +      * what is the job name?
 +      * com.seeburger.smarti.dao.DAOSysException: Unable to retrieve a database connection: javax.resource.ResourceException: IJ000460: Error checking for a transaction - caused by: com.seeburger.smarti.dao.DAOSysException: Unable to retrieve a database connection: javax.resource.ResourceException: IJ000460: Error checking for a transaction
 +
 +=== Conclusion ===
 +  * **[-]** FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte (see ''AttachmentDAOImpl.getFileAttachmentBinary()'') and it takes very long time. For 1GB file my computer needs ca. 50 minutes.
 +
 +==== Upload ====
 +
 +  * Is it possible to upload big files?
 +    * **No**
 +      * see ''AttachmentUploadHandler:77'' ->  ''this.baos = new ByteArrayOutputStream();'' -> This means the file will be fully loaded into the memory. Which makes working with large files impossible.
 +
 +
 +
 +
 +===== Import =====
 +In ''com.seeburger.airsupply.base.exchange.MimeMessageImportHandler'':
 +
 +<code java>
 +public void initializeHandlerObjects(SerializableInputStream in, MessageProperties props) {
 +    super.initializeHandlerObjects(in, props);
 +    this.mimeMessageImportProcessor = FATProcessorFactory.createMimeMessageImportProcessor(in, props, new XMLImportProviderImpl(),
 +        AttachmentProcessId.getIdBySmartiMType(props.getProperty(MessageProperties.MESSAGE_TYPE)));
 +}
 +
 +</code>
 +
 +Here we can use different ''mimeMessageImportProcessor'' depending on ''MESSAGE_TYPE''.
 +
 +
 +===== Misc =====
 +
 +==== Inserting BLOB for tests ====
 +
 +To insert BLOB create a new record in FAT META table at first:
 +
 +<code sql>
 +INSERT INTO TFATATTACHMENTMETA 
 +(CMIMECONTENTID,CPROCESSID,CFILENAME,CORIGINATOR,CFILESIZE,CFILETYPE,CENCRYPTED,CDELETEFLAG,CCREATIONTIMESTAMP,CTRANSMISSIONTIMESTAMP,CSCANSTATUS,CSCANMODE,CLASTSCANTIMESTAMP,CWSPROCESSORNODE,CVIRUSIDS,CRETRYBLOCK,CATTACHMENTID,CREADSTATUS,CREADTIMESTAMP,CFLEXFIELD1) 
 +VALUES 
 +('VVK_TEST_20190604',6,'attachment_1GB.txt','SUPPLIER',1048576000,'txt',0,0,1555482046306,-9223372036854775808,'STATUS_WATINGFORCHECK','ASYNC',0,null,null,0,3586,'NEW',-9223372036854775808,'COD');
 +</code>
 +
 +After that run the small program "Main" from my test project "insertFat" (see ''D:\development\workspaces\insertFat'').
 +
 +
 +===== Conclusion =====
 +
 +  * All steps should be checked first
 +    * To check:
 +      * JMS as technology
 +        * **Checked** -> It is not possible. 
 +          * JMS is not good technology to transfer big data.
 +          * BIS could not send/receive big files through JMS. (See email by Patrick Bruchhausen from 03.06.2019.)
 +      * Upload
 +        * **Checked** -> It is not possible at the moment.
 +      * FAT
 +        * **Checked** -> It is not possible at the moment. -> FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte
 +      * Antivirus 
 +        * Antivirus job
 +          * **Checked** -> It is not possible at the moment. -> FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte
 +        * Antivirus service
 +          * **TBD**
 +      * Download
 +        * **TBD**
 +      * ...
 +        * **TBD**
 +    * The time to test “all steps” should be planned.
 +  * After that all necessary fixes should be done
 +    * At least:
 +      * Replace JMS with other technology
 +        * Proposal: shared directory
 +      * FAT 
 +        * Read/Write files (don't read/write bytewise)
 +      * Upload
 +        * Read/write files (don't use ByteBufferedStream)