Skip to content

Large Attachements with MTOM and CXF

September 13, 2010

Recently i’ve solved a large file attachement problem in CXF web services using MTOM attachements. Using MTOM attachements and DataHanlder APIs I was successfully able to upload GBs of data by one user. First, lets talk about some of the background on the problem.

MTOM Overview

Binary data with SOAP messages is send in Base64 format because SAOP messages are based on plain text, so when binary data is converted into Base64 than its size is increased. And, in case of large attachement this kills the application and bring down the server with memory problems.

The solution with the large attachment with SOAP messages is the use of MTOM (SOAP Message Transmission Optimization Mechanism) which encodes binary data in base64Binary and sends the data as binary attachement rather than keeping it with actual SOAP message. MTOM is WC3 standard that is used to attach binary data with your SOAP messages. MTOM provides an elegant machenism to transfer binary data such as PDF, MS word, images, and other document types. MTOM uses XML-binary Optimized Packaging (XOP) packages for transmitting binary data.

MTOM Implementation

CXF provides support of MTOM with XOP implementation. To enable MTOM in you services you will need to define the element type in your WSDL as xsd:base64Binary for elements which will contain the binary data as shown in the following snippet:

	<s:complexType name="FileUpload">
		<s:attribute name="ByteData" type="s:base64Binary" use="optional"></s:attribute>
		<s:attribute name="Name" type="s:string"></s:attribute>
		<s:attribute name="Size" type="s:long" default="0"></s:attribute>
	</s:complexType>

In above example ByteData element of FileUpload is defined as the binary data. If you will convert the above with JAXB element you will get the following code:

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "FileUpload", propOrder = {
    "byteData"
})
public class FileUpload {

    @XmlAttribute(name = "ByteData")
    protected byte[] byteData;
   @XmlAttribute(name = "Name")
    protected String name;
    @XmlAttribute(name = "Size")
    protected Long size;

// getters and setters for above properties

Above code has defined binary data element as xsd:base64Binary and it is converted into a byte[] array but doesn’t take full advantage of MTOM optimization and all the binary data will be included in the actual SOAP message rather than serializing it.

To fully utilize the MTOM features you will need to use the xmime:expectedContentTypes attribute to your binary data elements with possible value of “application/octet-stream”. You can use other MIME type but “application/octet-stream” works in most cases. When you will add the “expectedContentTypes” attribute binary data will be send as an attachement and will not be included in the XML infoset. Following is the updated schema of FileUpload type which contains two elements with base64Binary but one with “expectedContentTypes”.

<s:schema xmlns:s="http://www.w3.org/2001/XMLSchema"
	targetNamespace="http://org.artstor.adam" xmlns:adam="http://org.artstor.adam"
	elementFormDefault="qualified" xmlns:xmime="http://www.w3.org/2005/05/xmlmime">

	<s:complexType name="FileUpload">
		<s:sequence>
			<s:element name="File" type="s:base64Binary"
				xmime:expectedContentTypes="application/octet-stream"></s:element>
		</s:sequence>
		<s:attribute name="ByteData" type="s:base64Binary" use="optional"></s:attribute>
		<s:attribute name="Name" type="s:string"></s:attribute>
		<s:attribute name="Size" type="s:long" default="0"></s:attribute>
	</s:complexType>

</s:schema>

If you will generate java classes for the above schema with JAXB you will get the following code. As you can see byteData properties is still using the byte[] but the second one that we have created with “expectedContentType” attribute has been created with a DataHandler.

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "FileUpload", propOrder = { "file" })
public class FileUpload {

	@XmlElement(name = "File", required = true)
	@XmlMimeType("application/octet-stream")
	protected DataHandler file;
	@XmlAttribute(name = "ByteData")
	protected byte[] byteData;
	@XmlAttribute(name = "Name")
	protected String name;
	@XmlAttribute(name = "Size")
	protected Long size;

// getters and setters for above properties
}

DataHandler and DataSource APIs does the actual streaming part of the MTOM attachements. There are different DataSource implementations available that you can use on the client side to initialize your POJOs to send the binary data to your web services. I will see how these can be used in the following section.

So, now we have done the schema part of the mtom attachement example. Lets, enable the MTOM feature.

MTOM Configuration

You can enable MTOM through java or configuration, i have used cxf.xml to enable mtom.

	  <jaxws:endpoint id="mtomService"  implementor="com.learn.cxf.mtom.MTOMServiceImpl" address="/soap">
	        <jaxws:properties>
	          <entry key="mtom-enabled" value="true"/>
	          <entry key="attachment-directory" value="/tmp/"/>
	          <entry key="attachment-memory-threshold" value="4000000"/>
	        </jaxws:properties>
	    </jaxws:endpoint>

As you can see i’ve added three properties to the jaxws:endpoint.

  1. mtom-enabled: This is the prime property to enable the MTOM, setting it true will enable MTOM feature.
  2. attachment-directory: This property is used to specify the directory to which binary data will be saved before streaming. This property is related to the next property and works along with the next property.
  3. attachement-memory-threshold: This property is use to set the memory threshold, that is use to keep the binary data in memory, data exceeding the memory threshold will be written to the directory specified by the attachement-directory property. Value of this property is set in bytes.

MTOM Client

The important part of MTOM Client in our example is the use of DataHandler and DataSource APIs to send the binary data to the web services.

I have used apache commons fileupload utility to process the multipart data. In the following example i’ve used our FileUpload class which is a java bean that will be used to upload MTOM attachements. I’ve also used fileupload utility with DiskFileItemFactory to retreive the multipart data from client. Following code extract shows the client code:


			FileItemFactory factory = new DiskFileItemFactory();
			ServletFileUpload upload = new ServletFileUpload(factory);

                        // we will upload multiple large files to our web service
			List<FileUpload> imgFileList = new ArrayList<FileUpload>();

				items = upload.parseRequest(request);
				for (FileItem fileItem : items) {
                                        // is it fileupload multipart field
					if (!fileItem.isFormField()) {
                                              FileUpload imgFile = new FileUpload();
                                              imgFile.setName(fileItem.getName());
                                              imgFile.setSize(fileItem.getSize());
                                              DataSource dataSource = new InputStreamDataSource(fileItem.getInputStream(), "application/octet-stream");
                                              DataHandler dataHandler = new DataHandler(dataSource);
                                              imgFile.setFile(dataHandler);
                                              imgFileList.add(imgFile);
                                       }
                             }

                    // call web service and forward uploaded images using web service request POJO
                    if(imgFileList.size() > 0) {
                          // UploadImageRequest is used as the input parameter to the uploadImage service
                          UploadImageRequest uploadImageRequest = new UploadImageRequest();
                          uploadImageRequest.getUploadedImages().addAll(imgFileList);
                          // service is an object of a web service port
                          service.uploadImages(uploadImageRequest);
                    }

Above code gives you an overview how you can use DataHanlder and InputStreamDataSource from client code to upload large data files. If you don’t want to use streaming and want to attach the whole data as a one byte array you could use ByteArrayDataSource. If you have file which is locally stored than you could also use FileDataSource.

In case of large files you will need to keep web service connection open until whole file is streamed and transfered to the web service. In order to change the connection timeout and enabling MTOM on client side using java API you will need to use the following code:


       Binding binding = ((BindingProvider)port).getBinding();
       ((SOAPBinding)binding).setMTOMEnabled(true);

       Client cl = ClientProxy.getClient(port);

       HTTPConduit http = (HTTPConduit) cl.getConduit();

       HTTPClientPolicy httpClientPolicy = new HTTPClientPolicy();
       // one hour timeout
       httpClientPolicy.setConnectionTimeout(1000 * 60 * 60 * 1);
       httpClientPolicy.setReceiveTimeout(1000 * 60 * 60 * 60 * 1);

       http.setClient(httpClientPolicy);

On web service end you can use DataHanlder.writeTo method to write all the streaming data to a file as shown below:

public void saveFiles(List<FileUpload> imgFileList) {
                for (FileUpload imageFile : imgFileList) {
                    if (imageFile.getFile() != null) {
                        FileOutputStream fos = null;
                        try{
                             fos = new FileOutputStream(destDir, new File(imageFile.getName());
                             imageFile.getFile().writeTo(fos);
                        }
                        catch(IOException ex) {
                           //TODO handle it
                        }
                        finally {
                            if (fos != null) {
                               try {
                                   fos.close();
                               } catch (IOException e) {
                                   //TODO handle it
                               }
                            }
                        }
                  }
}

In the end, i have written this article for intermediate programmers who can connect all the bits and peaces of the code pasted above as i have not given the complete example but enough information on how to use MTOM in CXF. If you want to read more on web services, don’t forget to subscribe to this blog!

Advertisements
2 Comments
  1. Thanks!

  2. jax-ws permalink

    Above code has defined binary data element as xsd:base64Binary and it is converted into a byte[] array but … and all the binary data will be included in the actual SOAP message rather than serializing it.

    I think this is not true, whether you specify as byte array or data handler, as long as MTOM is enabled, the binary data will be serialized and sent as attachment.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: