User Tools

Site Tools


airs-2113

AIRS-2113 ENABLER: Analyze - Handling large single processes (200 MB)

Common

Way

We can send information about links to upload/download files at least in two different ways:

Check full chain

  • The time to test the “full chain” should be planned.

Credentials

To download or upload attachments to / from a shared ressource we need credentials. Question:

  • how to get them?
    • from JBoss (standalone.xml or something like this)?

FAT

  • Can FAT work with big files?
    • At least DB column size (max. 2 GB) → should work, but test it first
  • Maybe it is something already available for sending/receiving links?

Antivirus

  • Can Antivirus work with big files?
    • Seems to work, because it seems that only the file name (and not payload) is transferred through the SOAP
    • - Test on DEV
  • Can the Antivirus job work with big files?
    • - test it locally
      • what is the job name?
      • com.seeburger.smarti.dao.DAOSysException: Unable to retrieve a database connection: javax.resource.ResourceException: IJ000460: Error checking for a transaction - caused by: com.seeburger.smarti.dao.DAOSysException: Unable to retrieve a database connection: javax.resource.ResourceException: IJ000460: Error checking for a transaction

Conclusion

  • [-] FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte (see AttachmentDAOImpl.getFileAttachmentBinary()) and it takes very long time. For 1GB file my computer needs ca. 50 minutes.

Upload

  • Is it possible to upload big files?
    • No
      • see AttachmentUploadHandler:77 → this.baos = new ByteArrayOutputStream(); → This means the file will be fully loaded into the memory. Which makes working with large files impossible.

Import

In com.seeburger.airsupply.base.exchange.MimeMessageImportHandler:

public void initializeHandlerObjects(SerializableInputStream in, MessageProperties props) {
    super.initializeHandlerObjects(in, props);
    this.mimeMessageImportProcessor = FATProcessorFactory.createMimeMessageImportProcessor(in, props, new XMLImportProviderImpl(),
        AttachmentProcessId.getIdBySmartiMType(props.getProperty(MessageProperties.MESSAGE_TYPE)));
}

Here we can use different mimeMessageImportProcessor depending on MESSAGE_TYPE.

Misc

Inserting BLOB for tests

To insert BLOB create a new record in FAT META table at first:

INSERT INTO TFATATTACHMENTMETA 
(CMIMECONTENTID,CPROCESSID,CFILENAME,CORIGINATOR,CFILESIZE,CFILETYPE,CENCRYPTED,CDELETEFLAG,CCREATIONTIMESTAMP,CTRANSMISSIONTIMESTAMP,CSCANSTATUS,CSCANMODE,CLASTSCANTIMESTAMP,CWSPROCESSORNODE,CVIRUSIDS,CRETRYBLOCK,CATTACHMENTID,CREADSTATUS,CREADTIMESTAMP,CFLEXFIELD1) 
VALUES 
('VVK_TEST_20190604',6,'attachment_1GB.txt','SUPPLIER',1048576000,'txt',0,0,1555482046306,-9223372036854775808,'STATUS_WATINGFORCHECK','ASYNC',0,NULL,NULL,0,3586,'NEW',-9223372036854775808,'COD');

After that run the small program “Main” from my test project “insertFat” (see D:\development\workspaces\insertFat).

Conclusion

  • All steps should be checked first
    • To check:
      • JMS as technology
        • Checked → It is not possible.
          • JMS is not good technology to transfer big data.
          • BIS could not send/receive big files through JMS. (See email by Patrick Bruchhausen from 03.06.2019.)
      • Upload
        • Checked → It is not possible at the moment.
      • FAT
        • Checked → It is not possible at the moment. → FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte
      • Antivirus
        • Antivirus job
          • Checked → It is not possible at the moment. → FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte
        • Antivirus service
          • TBD
      • Download
        • TBD
      • …
        • TBD
    • The time to test “all steps” should be planned.
  • After that all necessary fixes should be done
    • At least:
      • Replace JMS with other technology
        • Proposal: shared directory
      • FAT
        • Read/Write files (don't read/write bytewise)
      • Upload
        • Read/write files (don't use ByteBufferedStream)
airs-2113.txt · Last modified: by kalaschnikow