airs-2113
Table of Contents
AIRS-2113 ENABLER: Analyze - Handling large single processes (200 MB)
Common
Way
We can send information about links to upload/download files at least in two different ways:
- in special XML tags
- as MIME part of message from type
text/uri-list- more info:
Check full chain
- The time to test the “full chain” should be planned.
Credentials
To download or upload attachments to / from a shared ressource we need credentials. Question:
- how to get them?
- from JBoss (
standalone.xmlor something like this)?
FAT
- Can FAT work with big files?
- At least DB column size (max. 2 GB) → should work, but test it first
- Maybe it is something already available for sending/receiving links?
Antivirus
- Can Antivirus work with big files?
- Seems to work, because it seems that only the file name (and not payload) is transferred through the SOAP
- - Test on DEV
- Can the Antivirus job work with big files?
- - test it locally
- what is the job name?
- com.seeburger.smarti.dao.DAOSysException: Unable to retrieve a database connection: javax.resource.ResourceException: IJ000460: Error checking for a transaction - caused by: com.seeburger.smarti.dao.DAOSysException: Unable to retrieve a database connection: javax.resource.ResourceException: IJ000460: Error checking for a transaction
Conclusion
- [-] FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte (see
AttachmentDAOImpl.getFileAttachmentBinary()) and it takes very long time. For 1GB file my computer needs ca. 50 minutes.
Upload
- Is it possible to upload big files?
- No
- see
AttachmentUploadHandler:77→this.baos = new ByteArrayOutputStream();→ This means the file will be fully loaded into the memory. Which makes working with large files impossible.
Import
In com.seeburger.airsupply.base.exchange.MimeMessageImportHandler:
public void initializeHandlerObjects(SerializableInputStream in, MessageProperties props) { super.initializeHandlerObjects(in, props); this.mimeMessageImportProcessor = FATProcessorFactory.createMimeMessageImportProcessor(in, props, new XMLImportProviderImpl(), AttachmentProcessId.getIdBySmartiMType(props.getProperty(MessageProperties.MESSAGE_TYPE))); }
Here we can use different mimeMessageImportProcessor depending on MESSAGE_TYPE.
Misc
Inserting BLOB for tests
To insert BLOB create a new record in FAT META table at first:
INSERT INTO TFATATTACHMENTMETA (CMIMECONTENTID,CPROCESSID,CFILENAME,CORIGINATOR,CFILESIZE,CFILETYPE,CENCRYPTED,CDELETEFLAG,CCREATIONTIMESTAMP,CTRANSMISSIONTIMESTAMP,CSCANSTATUS,CSCANMODE,CLASTSCANTIMESTAMP,CWSPROCESSORNODE,CVIRUSIDS,CRETRYBLOCK,CATTACHMENTID,CREADSTATUS,CREADTIMESTAMP,CFLEXFIELD1) VALUES ('VVK_TEST_20190604',6,'attachment_1GB.txt','SUPPLIER',1048576000,'txt',0,0,1555482046306,-9223372036854775808,'STATUS_WATINGFORCHECK','ASYNC',0,NULL,NULL,0,3586,'NEW',-9223372036854775808,'COD');
After that run the small program “Main” from my test project “insertFat” (see D:\development\workspaces\insertFat).
Conclusion
- All steps should be checked first
- To check:
- JMS as technology
- Checked → It is not possible.
- JMS is not good technology to transfer big data.
- BIS could not send/receive big files through JMS. (See email by Patrick Bruchhausen from 03.06.2019.)
- Upload
- Checked → It is not possible at the moment.
- FAT
- Checked → It is not possible at the moment. → FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte
- Antivirus
- Antivirus job
- Checked → It is not possible at the moment. → FAT creates a temporary file to give it to antivirus, but it writes it bytewise = byte by byte
- Antivirus service
- TBD
- Download
- TBD
- …
- TBD
- The time to test “all steps” should be planned.
- After that all necessary fixes should be done
- At least:
- Replace JMS with other technology
- Proposal: shared directory
- FAT
- Read/Write files (don't read/write bytewise)
- Upload
- Read/write files (don't use ByteBufferedStream)
airs-2113.txt · Last modified: by kalaschnikow
